Re: [bitcoin-dev] Refreshed BIP324

From: Dhruv M <dhruv@bip324•com>
To: bitcoin-dev@lists•linuxfoundation.org
Subject: Re: [bitcoin-dev] Refreshed BIP324
Date: Thu, 16 Feb 2023 17:43:22 +0000	[thread overview]
Message-ID: <6b83c32e-59ca-65ef-2553-d66f8d117e56@bip324.com> (raw)
In-Reply-To: <Y7vMGVQz8TjS4Cad@erisian.com.au>

[-- Attachment #1: Type: text/plain, Size: 5923 bytes --]

Attempting to summarize the thread thus far:

Problem:

- 1 byte message type IDs are lacking a co-ordination mechanism when multiple in-flight BIPs are proposing new message types as the id space is reduced form 12 ASCII bytes to 1 byte.
- 1 byte IDs are scarce and should be allocated judiciously, especially given that gains on bandwidth are very much non-uniform across message types.

Solutions:

- Uniform encoding using the high-bit increases the available ID space drastically, however, there's still the issue of making sure that the most frequent message types get the shorter IDs.
- Making type IDs negotiable(editable, really) per direction per connection solves that issue at the cost of some increased complexity.

Since we don't really know the extent to which the protocol will ossify over time and that BIP324 is already quite a large change, we might want to optimize for the least additional complexity that doesn't close the doors on any of the solutions. How about this:

- BIP324 restricts type IDs to [1, 127]
- We remove 1 byte allocations for messages that are sent at most once per connection per direction
- Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern.

Best,

-Dhruv

On 1/9/23 00:11, Anthony Towns via bitcoin-dev wrote:

> On Fri, Jan 06, 2023 at 09:12:50AM +1000, Anthony Towns via bitcoin-dev wrote:
>
>> On Thu, Jan 05, 2023 at 10:06:29PM +0000, Pieter Wuille via bitcoin-dev wrote:
>>
>>> Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now.
>
>> The current implementation for 324 does the aliasing
>> as part of V2TransportDeserializer::GetMessage and
>> V2TransportSerializer::prepareForTransport. That makes a lot of sense,
>> [...]
>
> So I think you can make this setup work with a negotiated assignment of
> shortids, perhaps starting off something like:
> https://github.com/ajtowns/bitcoin/commit/6b8edd754bdcb582e293e4f5d0b41297711bdbb7
> That has a 242 element array per peer giving the mappings (which
> is just ~250 bytes per peer) for deserialization, which seems
> workable. [0]
>
> It also has a single global map for serialization, so we'll always shorten
> CFILTER to shortid 39 for every peer that supports shortids, even, eg, for
> a peer who's told us they'll send CFILTER as shortid 99 and that we should
> interpret shortid 39 from them as NEWFEATUREX. That has three advantages:
>
>  * each peer can choose a mapping that minimises their own outbound
>    traffic, even potentially for asymmetric connections, and don't need
>    to coordinate with the other peer to decide a common optimal mapping
>    that they both use across their connection
>
>  * you don't have to have different serialization tables per-peer,
>    reducing memory usage / implementation complexity
>
>  * you can leave V2TransportSerializer as a `const` object, and not have
>    to introduce additional locking logic to be able to update its
>    state...
>
> I'm not seeing a good way to introduce shortids for future one-shot
> negotiation messages though (like VERSION, VERACK, SENDADDRV2,
> WTXIDRELAY, SENDTXRCNCL):
>
>  * if you explicitly announce the mapping first, you're just wasting
>    bytes ("99=FOOBAR; 99 baz quux" vs just "FOOBAR baz quux")
>  * if you negotiate the tables you support between VERSION/VERACK and
>    then choose a mutually supported table after VERACK, that's too late
>    for pre-VERACK negotation messages
>  * announcing the tables you support as part of the VERSION message
>    would work, but seems a bit klunky
>
> Also, if you did want to shift to a new table, you'd probably want to
> always support sending/receiving {37, 44, 46, 47, 36} messages?
>
> I guess I still kind-of think it'd make more sense to just reserve
> shortids for post-VERACK messages that are going to be sent more
> than once per connection... At that point, even if you don't have any
> table in common with your peer, just following VERACK with an immediate
> announcement of each shortid you want to use and its meaning would still
> make reasonable sense.
>
> If we included the ability to define your own shortids concurrently
> with bip324 rollout, then I think nodes could always have a static set
> of shortids they use for all their peers for outbound messages, which,
> as above, seems like it would make for simpler implementations.
>
> ie, you might send:
>
>    VERSION
>    SHORTIDTBLS ["","awesomeshortids"]
>    WTXIDRELAY
>    SENDADDRV2
>    SENDPACKAGES 1
>    VERACK
>    SHORTID "" [(52,"getpkgtxns"), (53, "pkgtxns"), (54, "ancpkginfo")]
>
> ...but you'd do all that long form, and only switch to shortids for
> messages after you've declared exactly what your shortids are going to
> be.
>
> (where "" is the table name for bip324's table, and "awesomeshortids"
> is an updated table that includes the package relay commands already,
> perhaps)
>
> Cheers,
> aj
>
> [0] m_deserializer is used from the SocketHandler thread in
>     CNode::ReceiveMsgBytes(), but the p2p protocol is managed from the
>     MessageHandler thread; with multiple messages potentially deserialized
>     into vRecvMsg() at once -- but that means that if the first message
>     redefines shortid decoding, and the second message uses one of the
>     redefined shortids, it will have already been decoded incorrectly.
>     So that would need some futzing about still.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 7451 bytes --]