Hi Eric, aj, all,
Sorry for the delayed response. @aj I'm including some paraphrased points from our offline discussion (thanks).
> Other idea: what if you encode the parent txs as a short hash of the
wtxid (something like bip152 short ids? perhaps seeded per peer so
collisions will be different per peer?) and include that in the inv
announcement? Would that work to avoid a round trip almost all of the
time, while still giving you enough info to save bw by deduping parents?
> As I suggested earlier, a package is fundamentally a compact block (or
> block) announcement without the header. Compact block (BIP152) announcement
> is already well-defined and widely implemented...
> Let us not reinvent the wheel and/or introduce accidental complexity. I see
> no reason why packaging is not simply BIP152 without the 'header' field, an
> updated protocol version, and the following sort of changes to names
Interestingly, "why not use BIP 152 shortids to save bandwidth?" is by far the most common suggestion I hear (including offline feedback). Here's a full explanation:
BIP 152 shortens transaction hashes (32 bytes) to shortids (6 bytes) to save a significant amount of network bandwidth, which is extremely important in block relay. However, this comes at the expense of computational complexity. There is no way to directly calculate a transaction hash from a shortid; upon receipt of a compact block, a node is expected to calculate the shortids of every unconfirmed transaction it knows about to find the matches (BIP 152: [1], Bitcoin Core: [2]). This is expensive but appropriate for block relay, since the block must have a valid Proof of Work and new blocks only come every ~10 minutes. On the other hand, if we require nodes to calculate shortids for every transaction in their mempools every time they receive a package, we are creating a DoS vector. Unconfirmed transactions don't need PoW and, to have a live transaction relay network, we should expect nodes to handle transactions at a high-ish rate (i.e. at least 1000s of times more transactions than blocks). We can't pre-calculate or cache shortids for mempool transactions, since the SipHash key depends on the block hash and a per-connection salt.
Additionally, shortid calculation is not designed to prevent intentional individual collisions. If we were to use these shortids to deduplicate transactions we've supposedly already seen, we may have a censorship vector. Again, these tradeoffs make sense for compact block relay (see shortid section in BIP 152 [3]), but not package relay.
TLDR: DoSy if we calculate shortids on every package and censorship vector if we use shortids for deduplication.
> Given this message there is no reason
> to send a (potentially bogus) fee rate with every package. It can only be
> validated by obtaining the full set of txs, and the only recourse is
> dropping (etc.) the peer, as is the case with single txs.
Yeah, I agree with this. Combined with the previous discussion with aj (i.e. we can't accurately communicate the incentive compatibility of a package without sending the full graph, and this whole dance is to avoid downloading a few low-fee transactions in uncommon edge cases), I've realized I should remove the fee + weight information from pkginfo. Yay for less complexity!
Also, this might be pedantic, but I said something incorrect earlier and would like to correct myself:
>> In theory, yes, but maybe it was announced earlier (while our node was down?) or had dropped from our mempool or similar, either way we don't have those txs yet.
I said "It's fine if they have Erlay, since a sender would know in advance that B is missing and announce it as a package." But this isn't true since we're only using reconciliation in place of flooding to announce transactions as they arrive, not for rebroadcast, and we're not doing full mempool set reconciliation. In any case, making sure a node receives the transactions announced when it was offline is not something we guarantee, not an intended use case for package relay, and not worsened by this.
Thanks for your feedback!
Best,
Gloria