* [bitcoin-dev] Refreshed BIP324 @ 2022-10-08 12:59 Dhruv M 2022-10-26 16:39 ` Pieter Wuille 2023-10-11 20:52 ` Tim Ruffing 0 siblings, 2 replies; 22+ messages in thread From: Dhruv M @ 2022-10-08 12:59 UTC (permalink / raw) To: bitcoin-dev Hi all, We have refreshed the proposal for BIP324, a new bitcoin P2P protocol featuring opportunistic encryption, a mild bandwidth reduction, and the ability to negotiate upgrades before exchanging application messages. We'd like to invite community members to review the BIP[1] and the related Bitcoin Core code[2]. The proposal has a rich history[3]. The big changes since the last public appearance[4] are: * Elligator-swift encoding for the pubkeys in the ECDH exchange to obtain a pseudorandom bytestream * x-only ECDH secret derivation * Transport versioning that allows for upgradability * Trafic shapability using decoy packets and a shapable handshake * Complete rewrite of the BIP text We look forward to your review and comments. -Dhruv, Tim and Pieter [1] BIP Pull Request: https://github.com/bitcoin/bips/pull/1378 [2] All historical and current PRs: https://bip324.com/sections/code-review/ [3] https://bip324.com/sections/bip-review/ [4] https://gist.github.com/dhruv/5b1275751bc98f3b64bcafce7876b489 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-10-08 12:59 [bitcoin-dev] Refreshed BIP324 Dhruv M @ 2022-10-26 16:39 ` Pieter Wuille 2022-10-27 7:28 ` Vasil Dimov ` (2 more replies) 2023-10-11 20:52 ` Tim Ruffing 1 sibling, 3 replies; 22+ messages in thread From: Pieter Wuille @ 2022-10-26 16:39 UTC (permalink / raw) To: Dhruv M; +Cc: bitcoin-dev Hi all, On Saturday, October 8th, 2022 at 8:59 AM, Dhruv M <dhruv@bip324•com> wrote: > We have refreshed the proposal for BIP324, a new bitcoin P2P protocol > featuring opportunistic encryption, a mild bandwidth reduction, and the > ability > to negotiate upgrades before exchanging application messages. We'd like > to invite community members to review the BIP[1] and the related Bitcoin > Core > code[2]. One open question we have regarding BIP324's design is how to deal with the coordination of assigning the message type IDs. For context, the current BIP324 draft introduces a notion of 1-byte message type IDs, which take the place of the 12-byte command strings (in a backward compatible way; it's still possible to send full strings). This offers a mild bandwidth reduction (3 bytes per message overall), especially since many messages on the network are fairly small. However, it obviously raises the question of how the mapping table between the 1-byte IDs and the commands they represent should be maintained: 1. The most straightforward solution is using the BIP process as-is: let BIP324 introduce a fixed initial table, and future BIPs which introduce new messages can introduce new mapping entries for it. In theory, this is no worse than the current coordination difficulty about command strings, but in practice the risk of collisions due to competing proposals is of course significantly larger with 1-byte IDs vs. 12-byte strings. 2. An alternative approach is not using 1-byte IDs but slightly longer ones; for example 3-byte IDs, each consisting of a 2-byte BIP number and a 1-byte message index introduced by that BIP, at the cost of a smaller bandwidth improvement. This significantly reduces collision risks, but doesn't remove the coordination process concerns entirely (e.g. revisions changing what a BIP introduces need to be taken into account and probably still mean BIPs need to explicitly list which assignments they introduce). 3. Yet another possibility is not having a fixed table at all, and negotiate the mapping dynamically. E.g. either side could send a message at connection time with an explicit table of entries "when I send byte X, I mean command Y". 4. Lastly, the whole feature could just be dropped from BIP324 (sticking with command strings), and left for a follow-up (or independent) protocol improvement. Since arguably this is purely an application-layer concern and not a transport-layer one, it could even be added as an optional feature to the (pre-BIP324) protocol today. That would however very likely mean that BIP324 if adopted as-is isn't actually an (albeit small) bandwidth reduction compared to today, and forego a possibility to fix a fairly gratuitous inefficiency in the protocol from day one. Our idea is to start out with approach (1), with a mapping table effectively managed by the BIP process directly, but if and when collisions become a concern (maybe due to many parallel proposals, maybe because the number of messages just grows too big), switch to approach (3), possibly even differentially (the sent mappings are just additions/overwrites of the BIP-defined table mappings, rather than a full mapping). That said, we're not all that convinced this is the best approach, and feel this more a community/process question than a technical one, so it would be good to see more opinions on the topic. Cheers, -- Dhruv, Pieter, Tim ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-10-26 16:39 ` Pieter Wuille @ 2022-10-27 7:28 ` Vasil Dimov 2022-11-03 17:53 ` Murch 2022-11-08 3:20 ` Anthony Towns 2 siblings, 0 replies; 22+ messages in thread From: Vasil Dimov @ 2022-10-27 7:28 UTC (permalink / raw) To: Pieter Wuille, Bitcoin Protocol Discussion; +Cc: Dhruv M [-- Attachment #1: Type: text/plain, Size: 699 bytes --] On Wed, Oct 26, 2022 at 16:39:02 +0000, Pieter Wuille via bitcoin-dev wrote: [...] > Our idea is to start out with approach (1) [...] > That said, we're not all that convinced this is the best approach, and feel > this more a community/process question than a technical one, so it would be > good to see more opinions on the topic. [...] It all makes perfect sense to me. -- Vasil Dimov gro.DSBeerF@dv % If 10 years from now, when you are doing something quick and dirty, you suddenly visualize that I am looking over your shoulders and say to yourself, "Dijkstra would not have liked this", well that would be enough immortality for me. -- Edsger W. Dijkstra [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1528 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-10-26 16:39 ` Pieter Wuille 2022-10-27 7:28 ` Vasil Dimov @ 2022-11-03 17:53 ` Murch 2022-11-03 22:26 ` Jonas Schnelli 2022-11-08 3:20 ` Anthony Towns 2 siblings, 1 reply; 22+ messages in thread From: Murch @ 2022-11-03 17:53 UTC (permalink / raw) To: bitcoin-dev [-- Attachment #1.1: Type: text/plain, Size: 1303 bytes --] Hi Pieter, hello list, On 26.10.22 12:39, Pieter Wuille via bitcoin-dev wrote: > 1. The most straightforward solution is using the BIP process as-is: let BIP324 > introduce a fixed initial table, and future BIPs which introduce new > messages can introduce new mapping entries for it. In theory, this is no > worse than the current coordination difficulty about command strings, but > in practice the risk of collisions due to competing proposals is of course > significantly larger with 1-byte IDs vs. 12-byte strings. From what I understand we'll have about 35 message types on the network with the addition of BIP324. 256 possible IDs sounds like plenty room to grow, but perhaps we can be a bit more conservative: We could use the first bit to signal a 2-byte message ID. That allows us to express 128 IDs with 1 byte, but if we need more, we get a total of 2^15 IDs across 2 bytes. I would not be too concerned about collisions. Firstly, message types would probably be announced to the mailing list as part of the corresponding BIP, secondly, any overlooked collision should become apparent at implementation time. The risk could perhaps be further mitigated by encouraging less prevalent message types to use a 2-byte ID. Cheers, Murch [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-11-03 17:53 ` Murch @ 2022-11-03 22:26 ` Jonas Schnelli 0 siblings, 0 replies; 22+ messages in thread From: Jonas Schnelli @ 2022-11-03 22:26 UTC (permalink / raw) To: Murch, Bitcoin Protocol Discussion > From what I understand we'll have about 35 message types on the network with the addition of BIP324. 256 possible IDs sounds like plenty room to grow, but perhaps we can be a bit more conservative: > > We could use the first bit to signal a 2-byte message ID. That allows us to express 128 IDs with 1 byte, but if we need more, we get a total of 2^15 IDs across 2 bytes. Could make sense. There would be an alternative to preserve more 1 byte IDs on the cost of a (much) smaller 2 byte ID space: Reserve the short ID 0xFF as an indication for a 2 bytes short ID (additional 256 short IDs with 2 bytes). That could be done later outside BIP324. The 0xFF approach would lead to approx. 207 unused 1 byte short IDs (while Murchs approach would give us approx. 79 unused 1 byte short IDs). The signal bit two byte approach would however lead to ~32k more two byte message IDs. The main (and only?) benefit of short IDs is bandwidth. Short ID 1-12 are reserved for string based IDs and thus, new and rarely sent message types must not always use a short ID. Maybe the BIP should state that only frequent sent messages should reserve a short ID, though, the BIP itself assigns short IDs to all(?) message types (including low frequent messages like SENDHEADERS). Maybe exclude message types that expected to be only sent once from assigning a short ID? /j ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-10-26 16:39 ` Pieter Wuille 2022-10-27 7:28 ` Vasil Dimov 2022-11-03 17:53 ` Murch @ 2022-11-08 3:20 ` Anthony Towns 2022-11-10 21:23 ` Pieter Wuille 2 siblings, 1 reply; 22+ messages in thread From: Anthony Towns @ 2022-11-08 3:20 UTC (permalink / raw) To: Pieter Wuille, Bitcoin Protocol Discussion; +Cc: Dhruv M On Wed, Oct 26, 2022 at 04:39:02PM +0000, Pieter Wuille via bitcoin-dev wrote: > However, it obviously raises the question of how the mapping table between the > 1-byte IDs and the commands they represent should be maintained: > > 1. The most straightforward solution is using the BIP process as-is: let BIP324 > introduce a fixed initial table, and future BIPs which introduce new > messages can introduce new mapping entries for it. [...] > 3. Yet another possibility is not having a fixed table at all, and negotiate > the mapping dynamically. E.g. either side could send a message at > connection time with an explicit table of entries "when I send byte X, I > mean command Y". FWIW, I think these two options seem fine -- maintaining a purely local and hardcoded internal mapping of "message string C has id Y" where Y is capped by the number of commands you actually implement (presumably less than 65536 total) is easy, and providing a per-peer mapping from "byte X" to "id Y" then requires at most 512 bytes per peer, along with up to 3kB of initial setup to tell your peer what mappings you'll use. > Our idea is to start out with approach (1), with a mapping table effectively > managed by the BIP process directly, but if and when collisions become a > concern (maybe due to many parallel proposals, maybe because the number of > messages just grows too big), switch to approach (3), possibly even > differentially (the sent mappings are just additions/overwrites of the > BIP-defined table mappings, rather than a full mapping). I guess I think it would make sense to not start using a novel 1-byte message unless you've done something to introduce that message first; whether that's via approach (3) ("I'm going to use 0xE9 to mean pkgtxns") or via a multibyte feature support message ("I sent sendaddrv3 as a 10-byte message, that implies 0xA3 means addrv3 from now on"). I do still think it'd be better to recommend against reserving a byte for one-shot messages, and not do it for existing one-shot messages though. Cheers, aj ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-11-08 3:20 ` Anthony Towns @ 2022-11-10 21:23 ` Pieter Wuille 2022-11-12 3:23 ` Pieter Wuille 0 siblings, 1 reply; 22+ messages in thread From: Pieter Wuille @ 2022-11-10 21:23 UTC (permalink / raw) To: Bitcoin Protocol Discussion Hi, Thanks for the comments so far. I think these are all reasonable ideas. Comments inline below. On Thursday, November 3rd, 2022 at 1:53 PM, Murch wrote: > From what I understand we'll have about 35 message types on the network > with the addition of BIP324. 256 possible IDs sounds like plenty room to > grow, but perhaps we can be a bit more conservative: > > We could use the first bit to signal a 2-byte message ID. That allows us > to express 128 IDs with 1 byte, but if we need more, we get a total of > 2^15 IDs across 2 bytes. Yeah, effectively treating the first 1 or 2 bytes as a simple variable length integer is a nice way of increasing the space at low cost. This also doesn't need to be decided now. The initial approach could just be avoiding allocating bytes in the 128-255 range until the need for more space arises. If and when that is the case, the choice could be to: * Just continue treating the first byte as the command. * Start treating the first upper bit as a sign that another command byte follows. * Switch to some form of explicit signalling (option 3 is my earlier mail). On Thursday, November 3rd, 2022 at 6:26 PM, Jonas Schnelli wrote: > There would be an alternative to preserve more 1 byte IDs on the cost > of a (much) smaller 2 byte ID space: Reserve the short ID 0xFF as an > indication for a 2 bytes short ID (additional 256 short IDs with 2 bytes). I don't think this is needed, because we arguably already have that! If the first byte is 0x01, then 1 more command byte follows in the current BIP324 draft. That mechanism is designed for alphabetic 1-character commands, but nothing prevents it from also being used for other things (by using a non-alphabetic byte there). > Maybe the BIP should state that only frequent sent messages should reserve > a short ID, though, the BIP itself assigns short IDs to all(?) message > types (including low frequent messages like SENDHEADERS). > > Maybe exclude message types that expected to be only sent once from > assigning a short ID? I think that makes sense. Especially in combination with the idea avoiding bytes with the upper bit set there is a bit more pressure on the 1-byte space. Rarely-sent or at-most-once-sent commands don't really provide much benefit. I'd suggest scrapping from the list: * Version messages: version, verack * Negotiation messages: sendaddrv2, sendheaders, sendcmpct, wtxidrelay * Rarely-sent messages: mempool I'm not sure to what extent filteradd/filterload/filterclear/merkleblock are still actually used; perhaps they could be removed too? On Monday, November 7th, 2022 at 10:20 PM, Anthony Towns wrote: > I guess I think it would make sense to not start using a novel 1-byte > message unless you've done something to introduce that message first; > whether that's via approach (3) ("I'm going to use 0xE9 to mean pkgtxns") > or via a multibyte feature support message ("I sent sendaddrv3 as a > 10-byte message, that implies 0xA3 means addrv3 from now on"). That's fair, but I don't think it matters too much for allocation purposes; protocol designs should still not assign overlapping values, unless the protocols are known to never be used simultaneously? Unless... the assignment works like "whenever the sendaddrv3 is sent, the next available byte in range 48..127 gets allocated for addrv3". That means no explicit mapping is needed, as long as the total number of messages from simultaneously-active extensions isn't too large. > I do still think it'd be better to recommend against reserving a byte for > one-shot messages, and not do it for existing one-shot messages though. Agree. FWIW, if anyone was wondering about how much is actually saved by having 1-byte commands vs 12-byte commands, I've gathered statistics from two nodes (one with many inbound connections, one only outbound) for two weeks. This is obviously very dependent on network topology and local implementation choices, but it may still give an idea: * Outbound-only node: * Around 4.5% of sent bytes are bytes 2-12 of the command. * Sent 979.98 MiB in total. * Outbound and inbound node: * Around 1.6% of sent bytes are bytes 2-12 of the command. * Sent 124.14 GiB in total. Cheers, -- Pieter ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-11-10 21:23 ` Pieter Wuille @ 2022-11-12 3:23 ` Pieter Wuille 2022-11-12 18:52 ` Yuval Kogman 2022-11-18 8:24 ` Anthony Towns 0 siblings, 2 replies; 22+ messages in thread From: Pieter Wuille @ 2022-11-12 3:23 UTC (permalink / raw) To: Bitcoin Protocol Discussion Another idea... On Thursday, November 10th, 2022 at 4:23 PM, Pieter Wuille via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > On Thursday, November 3rd, 2022 at 1:53 PM, Murch wrote: > > > From what I understand we'll have about 35 message types on the network > > with the addition of BIP324. 256 possible IDs sounds like plenty room to > > grow, but perhaps we can be a bit more conservative: > > > > We could use the first bit to signal a 2-byte message ID. That allows us > > to express 128 IDs with 1 byte, but if we need more, we get a total of > > 2^15 IDs across 2 bytes. > > Yeah, effectively treating the first 1 or 2 bytes as a simple variable > length integer is a nice way of increasing the space at low cost. The above would really result in having two separate variable-length encodings: * First byte 1-12 to signify length of alphabetic command * Otherwise first bit to signify length of short command I think we can just merge the two and have a single variable-length command structure that can be used for both: command encodings are 1 to 12 bytes, each byte's top bit indicating whether another byte follows (the top bit of byte 11 has no special meaning). This means: * Every alphabetic command of L characters becomes L bytes. * 102 non-alphabetic 1-byte commands can be assigned. * 15708 non-alphabetic 2-byte commands can be assigned. * etc So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1). WDYT? Cheers, -- Pieter ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-11-12 3:23 ` Pieter Wuille @ 2022-11-12 18:52 ` Yuval Kogman 2022-11-18 8:24 ` Anthony Towns 1 sibling, 0 replies; 22+ messages in thread From: Yuval Kogman @ 2022-11-12 18:52 UTC (permalink / raw) To: Pieter Wuille, Bitcoin Protocol Discussion [-- Attachment #1: Type: text/plain, Size: 1574 bytes --] On Sat, 12 Nov 2022, 11:01 Pieter Wuille via bitcoin-dev, < bitcoin-dev@lists•linuxfoundation.org> wrote: > I think we can just merge the two and have a single variable-length > command structure that can be used for both: command encodings are 1 to 12 > bytes, each byte's top bit indicating whether another byte follows (the top > bit of byte 11 has no special meaning). > ... > So this gives a uniform space which commands can be assigned from, and > there is no strict need for thinking of the short-binary and > long-alphabetic commands as distinct.In v2, some short ones would be > treated as aliases for old long-alphabetic ones. This seems a bit dubious, but since command names are ASCII strings reversing the meaning of the top bit so that 0 signifies a following byte would allow the alphabetic commands to be reinterpreted as non-alphabetic, so the space no longer needs to be a disjoint union of two sub-spaces which arguably takes the "no [...] need for thinking of [them] as distinct" logic a little bit bit farther. The main downsides are that this does nothing to re-assign shorter codes to existing commands, and secondly that even the non-alphabetic code path completely supersedes the alphabetic one, the numerical values are up to 85 or 86 bits wide, which is not a reasonable word size. Perhaps this is not a concern since as far as I know there are no collisions in the first 9 bytes of existing commands, and constrain the non-alphabetic representation to 9 bytes would leave the last top bit free, so the space would be isomorphic to {0,1}^64. [-- Attachment #2: Type: text/html, Size: 2201 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-11-12 3:23 ` Pieter Wuille 2022-11-12 18:52 ` Yuval Kogman @ 2022-11-18 8:24 ` Anthony Towns 2023-01-05 22:06 ` Pieter Wuille 1 sibling, 1 reply; 22+ messages in thread From: Anthony Towns @ 2022-11-18 8:24 UTC (permalink / raw) To: Pieter Wuille, Bitcoin Protocol Discussion On Sat, Nov 12, 2022 at 03:23:16AM +0000, Pieter Wuille via bitcoin-dev wrote: > Another idea... > This means: > * Every alphabetic command of L characters becomes L bytes. > * 102 non-alphabetic 1-byte commands can be assigned. > * 15708 non-alphabetic 2-byte commands can be assigned. (There are 128**L possible L-byte commands, 26**L alphabetic L-byte commands, and hence 128**L-26**L non-alphabetical L-byte commands) > * etc > So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1). Isn't that optimising for the wrong thing? Aren't the goals we want: 1) it should be easy to come up with a message identifier without accidently conflicting with someone else's proposal 2) commonly used messages on the wire should have a short encoding in order to save bandwidth Depending on how much the p2p protocol ossifies, which messages are "commonly used on the wire" might be expected to change; and picking an otherwise meaningless value from a set of 102 elements seems likely to produce conflicts... Doing: -> VERSION <- VERSION <- SHORTMSG ["BIP324", "foo.org/experiment"] <- VERACK -> SHORTMSG ["BIP324"] -> VERACK -> SENDSHORTMSG "BIP324" [(13, "ADDRV3")] <- SENDSHORTMSG "BIP324" -> 34 (SENDHEADERS) -> 25 (GETHEADERS) ... where "SHORTMSG" lets you specify an array of tables you support (such as BIP324's), and, once you know both sides supports a table, you can send `SENDSHORTMSG` to choose the table you'll use to abbreviate messages you send, as well as any modifications to that table (like replacing ADDR with ADDRV3). Then when writing BIPs, you just choose a meaningful string ("ADDRV3"), and when implementing you send a one-time "SENDSHORTMSG" message to optimise the messages you'll send most often... As time goes on and the most common messages change, issue a new BIP with a new table so that your one-time SHORTIDs message becomes shorter too, at least when you connect to peers that also know about the new bip.. Could potentially support that without bip324, by taking over the one byte message space, presuming you don't have any one-character messages you actually want to send? Could do this /as well as/ the encoding above, I guess; then you would have bips specify alphabetic message ids, and use SENDSHORTMSG to dynamically (and sequentially) override/allocate non-alphabetic ids? That would presumably limit the number of non-alphabetic ids to how many you could specify in a single SENDSHORTIDs message, which I think would be up to something like 749k different 1/2/3/4/5/6-byte alphabetic message ids (encoded as 1/2/3-byte non-alphabetic messages). Probably some more specific limit would be better though. Cheers, aj ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-11-18 8:24 ` Anthony Towns @ 2023-01-05 22:06 ` Pieter Wuille 2023-01-05 23:12 ` Anthony Towns 0 siblings, 1 reply; 22+ messages in thread From: Pieter Wuille @ 2023-01-05 22:06 UTC (permalink / raw) To: Anthony Towns; +Cc: Bitcoin Protocol Discussion ------- Original Message ------- On Friday, November 18th, 2022 at 3:24 AM, Anthony Towns <aj@erisian•com.au> wrote: > > * etc > > So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1). > > Isn't that optimising for the wrong thing? Aren't the goals we want: > > 1) it should be easy to come up with a message identifier without > accidently conflicting with someone else's proposal > > 2) commonly used messages on the wire should have a short encoding > in order to save bandwidth > > Depending on how much the p2p protocol ossifies, which messages are > "commonly used on the wire" might be expected to change; and picking an > otherwise meaningless value from a set of 102 elements seems likely to > produce conflicts... Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now. I just thought it would be interesting to have a uniform encoding without explicit distinction between "short commands" and "long commands" at that layer. But maybe none of this is worth it, as it's perhaps more complexity than the alternative, and the alternative already has a working implementation and written-up specification. Cheers, -- Pieter ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-01-05 22:06 ` Pieter Wuille @ 2023-01-05 23:12 ` Anthony Towns 2023-01-09 8:11 ` Anthony Towns 0 siblings, 1 reply; 22+ messages in thread From: Anthony Towns @ 2023-01-05 23:12 UTC (permalink / raw) To: Pieter Wuille, Bitcoin Protocol Discussion On Thu, Jan 05, 2023 at 10:06:29PM +0000, Pieter Wuille via bitcoin-dev wrote: > > > So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1). > Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now. > > I just thought it would be interesting to have a uniform encoding without explicit distinction between "short commands" and "long commands" at that layer. > But maybe none of this is worth it, as it's perhaps more complexity than the alternative, and the alternative already has a working implementation and written-up specification. Heh, I was just looking at this yesterday, but failing to quite reach a conclusion. One thing I hadn't realised about this was that it's not actually a restriction compared to what we currently allow with p2p v1: CMessageHeader::IsCommandValid() already rejects commands that use characters outside of 0x20 to 0x7E, so the high bit is already available for signalling when we reach the last byte. The current implementation for 324 does the aliasing as part of V2TransportDeserializer::GetMessage and V2TransportSerializer::prepareForTransport. That makes a lot of sense, but particularly if we were to negotiate short commands sometime around VERSION or VERACK, it might make more sense for the aliasing to move up to the protocol layer rather than have it close to the wire layer. In that case having a uniform encoding means we could just keep using CSerializedNetMsg whether we're sending a short command or a multibyte ascii command -- without a uniform encoding, if we wanted to move short commands up a layer, I think we'd need to change CSerializedNetMsg to have m_type be a `std::variant<uint8_t,std::string>` instead of just a string, or something similar. I think I'm leaning towards "it doesn't matter either way" though: * if we can negotiate short commands on a per-peer basis, then once negotiation's finished we'll only be using short commands so saving a byte on long commands doesn't matter much * if we've only got around 30 or 40 commands we understand anyway (even counting one-time-only negotiation stuff), then it doesn't matter if we can do 102, 126 or 242 short commands since those are all more than we need * whether we'd have to tweak an internal struct if we want to change the way our code is structured shouldn't really be much of an influence on protocol design... Cheers, aj ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-01-05 23:12 ` Anthony Towns @ 2023-01-09 8:11 ` Anthony Towns 2023-02-16 17:43 ` Dhruv M 0 siblings, 1 reply; 22+ messages in thread From: Anthony Towns @ 2023-01-09 8:11 UTC (permalink / raw) To: Bitcoin Protocol Discussion On Fri, Jan 06, 2023 at 09:12:50AM +1000, Anthony Towns via bitcoin-dev wrote: > On Thu, Jan 05, 2023 at 10:06:29PM +0000, Pieter Wuille via bitcoin-dev wrote: > > Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now. > The current implementation for 324 does the aliasing > as part of V2TransportDeserializer::GetMessage and > V2TransportSerializer::prepareForTransport. That makes a lot of sense, > [...] So I think you can make this setup work with a negotiated assignment of shortids, perhaps starting off something like: https://github.com/ajtowns/bitcoin/commit/6b8edd754bdcb582e293e4f5d0b41297711bdbb7 That has a 242 element array per peer giving the mappings (which is just ~250 bytes per peer) for deserialization, which seems workable. [0] It also has a single global map for serialization, so we'll always shorten CFILTER to shortid 39 for every peer that supports shortids, even, eg, for a peer who's told us they'll send CFILTER as shortid 99 and that we should interpret shortid 39 from them as NEWFEATUREX. That has three advantages: * each peer can choose a mapping that minimises their own outbound traffic, even potentially for asymmetric connections, and don't need to coordinate with the other peer to decide a common optimal mapping that they both use across their connection * you don't have to have different serialization tables per-peer, reducing memory usage / implementation complexity * you can leave V2TransportSerializer as a `const` object, and not have to introduce additional locking logic to be able to update its state... I'm not seeing a good way to introduce shortids for future one-shot negotiation messages though (like VERSION, VERACK, SENDADDRV2, WTXIDRELAY, SENDTXRCNCL): * if you explicitly announce the mapping first, you're just wasting bytes ("99=FOOBAR; 99 baz quux" vs just "FOOBAR baz quux") * if you negotiate the tables you support between VERSION/VERACK and then choose a mutually supported table after VERACK, that's too late for pre-VERACK negotation messages * announcing the tables you support as part of the VERSION message would work, but seems a bit klunky Also, if you did want to shift to a new table, you'd probably want to always support sending/receiving {37, 44, 46, 47, 36} messages? I guess I still kind-of think it'd make more sense to just reserve shortids for post-VERACK messages that are going to be sent more than once per connection... At that point, even if you don't have any table in common with your peer, just following VERACK with an immediate announcement of each shortid you want to use and its meaning would still make reasonable sense. If we included the ability to define your own shortids concurrently with bip324 rollout, then I think nodes could always have a static set of shortids they use for all their peers for outbound messages, which, as above, seems like it would make for simpler implementations. ie, you might send: VERSION SHORTIDTBLS ["","awesomeshortids"] WTXIDRELAY SENDADDRV2 SENDPACKAGES 1 VERACK SHORTID "" [(52,"getpkgtxns"), (53, "pkgtxns"), (54, "ancpkginfo")] ...but you'd do all that long form, and only switch to shortids for messages after you've declared exactly what your shortids are going to be. (where "" is the table name for bip324's table, and "awesomeshortids" is an updated table that includes the package relay commands already, perhaps) Cheers, aj [0] m_deserializer is used from the SocketHandler thread in CNode::ReceiveMsgBytes(), but the p2p protocol is managed from the MessageHandler thread; with multiple messages potentially deserialized into vRecvMsg() at once -- but that means that if the first message redefines shortid decoding, and the second message uses one of the redefined shortids, it will have already been decoded incorrectly. So that would need some futzing about still. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-01-09 8:11 ` Anthony Towns @ 2023-02-16 17:43 ` Dhruv M 2023-02-17 15:51 ` Anthony Towns 0 siblings, 1 reply; 22+ messages in thread From: Dhruv M @ 2023-02-16 17:43 UTC (permalink / raw) To: bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 5923 bytes --] Attempting to summarize the thread thus far: Problem: - 1 byte message type IDs are lacking a co-ordination mechanism when multiple in-flight BIPs are proposing new message types as the id space is reduced form 12 ASCII bytes to 1 byte. - 1 byte IDs are scarce and should be allocated judiciously, especially given that gains on bandwidth are very much non-uniform across message types. Solutions: - Uniform encoding using the high-bit increases the available ID space drastically, however, there's still the issue of making sure that the most frequent message types get the shorter IDs. - Making type IDs negotiable(editable, really) per direction per connection solves that issue at the cost of some increased complexity. Since we don't really know the extent to which the protocol will ossify over time and that BIP324 is already quite a large change, we might want to optimize for the least additional complexity that doesn't close the doors on any of the solutions. How about this: - BIP324 restricts type IDs to [1, 127] - We remove 1 byte allocations for messages that are sent at most once per connection per direction - Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern. Best, -Dhruv On 1/9/23 00:11, Anthony Towns via bitcoin-dev wrote: > On Fri, Jan 06, 2023 at 09:12:50AM +1000, Anthony Towns via bitcoin-dev wrote: > >> On Thu, Jan 05, 2023 at 10:06:29PM +0000, Pieter Wuille via bitcoin-dev wrote: >> >>> Oh, yes. I meant this as an encoding scheme, not as a (replacement for) the negotiation/coordination mechanism. There could still be an initial assignment for 1-byte encodings, and/or an explicit mechanism to negotiate other assignment, and/or nothing at all for now. > >> The current implementation for 324 does the aliasing >> as part of V2TransportDeserializer::GetMessage and >> V2TransportSerializer::prepareForTransport. That makes a lot of sense, >> [...] > > So I think you can make this setup work with a negotiated assignment of > shortids, perhaps starting off something like: > https://github.com/ajtowns/bitcoin/commit/6b8edd754bdcb582e293e4f5d0b41297711bdbb7 > That has a 242 element array per peer giving the mappings (which > is just ~250 bytes per peer) for deserialization, which seems > workable. [0] > > It also has a single global map for serialization, so we'll always shorten > CFILTER to shortid 39 for every peer that supports shortids, even, eg, for > a peer who's told us they'll send CFILTER as shortid 99 and that we should > interpret shortid 39 from them as NEWFEATUREX. That has three advantages: > > * each peer can choose a mapping that minimises their own outbound > traffic, even potentially for asymmetric connections, and don't need > to coordinate with the other peer to decide a common optimal mapping > that they both use across their connection > > * you don't have to have different serialization tables per-peer, > reducing memory usage / implementation complexity > > * you can leave V2TransportSerializer as a `const` object, and not have > to introduce additional locking logic to be able to update its > state... > > I'm not seeing a good way to introduce shortids for future one-shot > negotiation messages though (like VERSION, VERACK, SENDADDRV2, > WTXIDRELAY, SENDTXRCNCL): > > * if you explicitly announce the mapping first, you're just wasting > bytes ("99=FOOBAR; 99 baz quux" vs just "FOOBAR baz quux") > * if you negotiate the tables you support between VERSION/VERACK and > then choose a mutually supported table after VERACK, that's too late > for pre-VERACK negotation messages > * announcing the tables you support as part of the VERSION message > would work, but seems a bit klunky > > Also, if you did want to shift to a new table, you'd probably want to > always support sending/receiving {37, 44, 46, 47, 36} messages? > > I guess I still kind-of think it'd make more sense to just reserve > shortids for post-VERACK messages that are going to be sent more > than once per connection... At that point, even if you don't have any > table in common with your peer, just following VERACK with an immediate > announcement of each shortid you want to use and its meaning would still > make reasonable sense. > > If we included the ability to define your own shortids concurrently > with bip324 rollout, then I think nodes could always have a static set > of shortids they use for all their peers for outbound messages, which, > as above, seems like it would make for simpler implementations. > > ie, you might send: > > VERSION > SHORTIDTBLS ["","awesomeshortids"] > WTXIDRELAY > SENDADDRV2 > SENDPACKAGES 1 > VERACK > SHORTID "" [(52,"getpkgtxns"), (53, "pkgtxns"), (54, "ancpkginfo")] > > ...but you'd do all that long form, and only switch to shortids for > messages after you've declared exactly what your shortids are going to > be. > > (where "" is the table name for bip324's table, and "awesomeshortids" > is an updated table that includes the package relay commands already, > perhaps) > > Cheers, > aj > > [0] m_deserializer is used from the SocketHandler thread in > CNode::ReceiveMsgBytes(), but the p2p protocol is managed from the > MessageHandler thread; with multiple messages potentially deserialized > into vRecvMsg() at once -- but that means that if the first message > redefines shortid decoding, and the second message uses one of the > redefined shortids, it will have already been decoded incorrectly. > So that would need some futzing about still. > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [-- Attachment #2: Type: text/html, Size: 7451 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-02-16 17:43 ` Dhruv M @ 2023-02-17 15:51 ` Anthony Towns 2023-02-17 22:13 ` Pieter Wuille 0 siblings, 1 reply; 22+ messages in thread From: Anthony Towns @ 2023-02-17 15:51 UTC (permalink / raw) To: Dhruv M, Bitcoin Protocol Discussion On Thu, Feb 16, 2023 at 05:43:22PM +0000, Dhruv M via bitcoin-dev wrote: > Problem: > - 1 byte message type IDs are lacking a co-ordination mechanism when multiple in-flight BIPs are proposing new message types as the id space is reduced form 12 ASCII bytes to 1 byte. > - 1 byte IDs are scarce and should be allocated judiciously, especially given that gains on bandwidth are very much non-uniform across message types. ACK. > Solutions: > - Uniform encoding using the high-bit increases the available ID space drastically, however, there's still the issue of making sure that the most frequent message types get the shorter IDs. > - Making type IDs negotiable(editable, really) per direction per connection solves that issue at the cost of some increased complexity. > > Since we don't really know the extent to which the protocol will ossify over time and that BIP324 is already quite a large change, we might want to optimize for the least additional complexity that doesn't close the doors on any of the solutions. I think it's probably less complex to close *some* of the doors? In particular, I think there's two questions that have to get answered: 1) how do you distinguish the command from the payload for non short-ids -- by a length prefix, or by setting the high-bit of the final command byte? 2) are short ids available/meaningful to send prior to VERACK being completed? > How about this: > - BIP324 restricts type IDs to [1, 127] Is this for short ids (currently [13-255] per the bip) or for every byte in a non-short-id command (for p2p v1, IsCommandValid() restricts each byte to being in the printable ascii range, ie [32-126])? Here's another approach: idea: we use short ids to minimise bandwidth, and don't care about bandwidth for long ids implementation: short id 0 is reserved for long commands. when received, we decode the first 12 bytes of the payload and treat them exactly the same as a v1 p2p message (trailing 0-bytes, etc) (if there's not 12 bytes of payload, it's just treated as an invalid command and dropped) short ids 1-255 are available for use as aliases of particular long commands (That's exactly compatible with p2p v1, and also avoids the temptation to try to choose short command names rather than descriptive ones -- the 0-padding to 12 bytes prevents you from saving any bandwidth that way; but that's what we have short ids for anyway) If we decide we want >255 short ids, we can figure out how to extend them later, in a fairly open ended way I think, eg by having [128-255] imply a 2 byte short id, so that seems fine? > - We remove 1 byte allocations for messages that are sent at most once per connection per direction I think this leaves 32 commands that get short ids initially: misc: ADDR, ADDRV2, BLOCK, FEEFILTER, GETBLOCKS, GETDATA, GETHEADERS, HEADERS, INV, NOTFOUND, PING, PONG, TX bip 35/37: FILTERADD, FILTERCLEAR, FILTERLOAD, MEMPOOL, MERKLEBLOCK bip 152: BLOCKTXN, CMPCTBLOCK, GETBLOCKTXN bip 157: CFCHECKPT, CFHEADERS, CFILTER, GETCFCHCKPT, GETCFHEADERS, GETCFILTERS bip 330: RECONCILDIFF, REQRECON, REQSKETCHEXT, SENDCMPCT, SKETCH which drops: VERSION, VERACK, GETADDR, SENDADDRV2, SENDHEADERS, SENDTXRCNCL, WTXIDRELAY compared to bip 324 currently. I think the things missing from the current list (and not currently in use by bitcoin core) are: bip 61: REJECT bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO > - Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern. I agree that's fine, though I expect that we'll probably want to do it not long after bip 331 is ready for merge (or some other p2p improvement comes along)... Cheers, aj ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-02-17 15:51 ` Anthony Towns @ 2023-02-17 22:13 ` Pieter Wuille 2023-02-19 23:56 ` Anthony Towns 0 siblings, 1 reply; 22+ messages in thread From: Pieter Wuille @ 2023-02-17 22:13 UTC (permalink / raw) To: Anthony Towns, Bitcoin Protocol Discussion; +Cc: Dhruv M On Friday, February 17th, 2023 at 10:51 AM, Anthony Towns via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > I think it's probably less complex to close some of the doors? > 2) are short ids available/meaningful to send prior to VERACK being > completed? Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for *re*-negotiating. > Here's another approach: > > idea: we use short ids to minimise bandwidth, and don't care about > bandwidth for long ids > > implementation: > short id 0 is reserved for long commands. when received, we > decode the first 12 bytes of the payload and treat them > exactly the same as a v1 p2p message (trailing 0-bytes, etc) > (if there's not 12 bytes of payload, it's just treated as an > invalid command and dropped) > > short ids 1-255 are available for use as aliases of particular > long commands > > (That's exactly compatible with p2p v1, and also avoids the temptation > to try to choose short command names rather than descriptive ones -- the > 0-padding to 12 bytes prevents you from saving any bandwidth that way; > but that's what we have short ids for anyway) I like this idea. It avoids the variable-length encoding question and related complexity entirely for things where we admittedly don't care about the bandwidth impact anyway. It may also have another (rather weak) advantage, in that it may reduce how much information a passive observe may learn about application level features (sendheaders, sendaddrv2, ...) from the packet size sent (which would otherwise depend on command lengths), even when decoys are not in use, if no short commands are included for these messages. > > - We remove 1 byte allocations for messages that are sent at most once per connection per direction > > I think this leaves 32 commands that get short ids initially: > > misc: ADDR, ADDRV2, BLOCK, FEEFILTER, GETBLOCKS, GETDATA, GETHEADERS, > HEADERS, INV, NOTFOUND, PING, PONG, TX > bip 35/37: FILTERADD, FILTERCLEAR, FILTERLOAD, MEMPOOL, MERKLEBLOCK > bip 152: BLOCKTXN, CMPCTBLOCK, GETBLOCKTXN > bip 157: CFCHECKPT, CFHEADERS, CFILTER, GETCFCHCKPT, GETCFHEADERS, > GETCFILTERS > bip 330: RECONCILDIFF, REQRECON, REQSKETCHEXT, SENDCMPCT, SKETCH Sounds right. > which drops: > > VERSION, VERACK, GETADDR, SENDADDRV2, SENDHEADERS, SENDTXRCNCL, > WTXIDRELAY Indeed. > compared to bip 324 currently. > > I think the things missing from the current list (and not currently in > use by bitcoin core) are: > > bip 61: REJECT > bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO Do you feel REJECT should be included? > > - Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern. > > I agree that's fine, though I expect that we'll probably want to do it > not long after bip 331 is ready for merge (or some other p2p improvement > comes along)... I do prefer that as well; it feels like the transport layer shouldn't be aware of the different command names that exist, but this is very much just an implementation issue. Perhaps a possibility is having the transport layer translate short-command-number-N to the 12-byte command "\x00\x00..." + byte(N), and hand that to the application layer, which could then do the mapping? Cheers, -- Pieter ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-02-17 22:13 ` Pieter Wuille @ 2023-02-19 23:56 ` Anthony Towns 2023-02-20 15:22 ` Pieter Wuille 0 siblings, 1 reply; 22+ messages in thread From: Anthony Towns @ 2023-02-19 23:56 UTC (permalink / raw) To: Pieter Wuille, Bitcoin Protocol Discussion; +Cc: Dhruv M On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via bitcoin-dev wrote: > > I think it's probably less complex to close some of the doors? > > 2) are short ids available/meaningful to send prior to VERACK being > > completed? > Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for *re*-negotiating. I think you still need/want two negotiation steps -- once to tell each other what tables you know about, once to choose a mutually recognised table and specify any additions. > > I think the things missing from the current list (and not currently in > > use by bitcoin core) are: > > bip 61: REJECT > > bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO > Do you feel REJECT should be included? I don't think it matters much; reject messages are both rare and include a reason so you'd only be saving maybe 12 bytes out of 62 (~20%) for maybe 6000 messages a day per peer that sends reject messages, so 72kB/day/reject-peer? > Perhaps a possibility is having the transport layer translate short-command-number-N to the 12-byte command "\x00\x00..." + byte(N), and hand that to the application layer, which could then do the mapping? Presuming the transport layer also continues to reject commands that have a '\x00' byte at the start or in the middle (ie !IsCommandValid()), that seems pretty reasonable... Cheers, aj ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-02-19 23:56 ` Anthony Towns @ 2023-02-20 15:22 ` Pieter Wuille 2023-02-21 16:03 ` Anthony Towns 0 siblings, 1 reply; 22+ messages in thread From: Pieter Wuille @ 2023-02-20 15:22 UTC (permalink / raw) To: Anthony Towns; +Cc: Bitcoin Protocol Discussion, Dhruv M On Sunday, February 19th, 2023 at 6:56 PM, Anthony Towns <aj@erisian•com.au> wrote: > On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via bitcoin-dev wrote: > > > > I think it's probably less complex to close some of the doors? > > > 2) are short ids available/meaningful to send prior to VERACK being > > > completed? > > > Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for re-negotiating. > > I think you still need/want two negotiation steps -- once to tell each > other what tables you know about, once to choose a mutually recognised > table and specify any additions. Right, I wasn't talking about how many steps/messages the negotiation takes. I just meant that if all negotiation of the mapping table happens just once (before VERACK) and that negotiation itself happens without use of short commands, then there is no need for re-negotiating short commands after they are already in use. Nothing concrete, but I can imagine that that may simplify some implementations. Cheers, -- Pieter ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-02-20 15:22 ` Pieter Wuille @ 2023-02-21 16:03 ` Anthony Towns 2023-02-28 18:07 ` Dhruv M 0 siblings, 1 reply; 22+ messages in thread From: Anthony Towns @ 2023-02-21 16:03 UTC (permalink / raw) To: Pieter Wuille, Bitcoin Protocol Discussion; +Cc: Dhruv M On Mon, Feb 20, 2023 at 03:22:30PM +0000, Pieter Wuille via bitcoin-dev wrote: > On Sunday, February 19th, 2023 at 6:56 PM, Anthony Towns <aj@erisian•com.au> wrote: > > On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via bitcoin-dev wrote: > > > > I think it's probably less complex to close some of the doors? > > > > 2) are short ids available/meaningful to send prior to VERACK being > > > > completed? > > > > Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for re-negotiating. > > I think you still need/want two negotiation steps -- once to tell each > > other what tables you know about, once to choose a mutually recognised > > table and specify any additions. > Right, I wasn't talking about how many steps/messages the negotiation takes. I just meant that if all negotiation of the mapping table happens just once (before VERACK) and that negotiation itself happens without use of short commands, then there is no need for re-negotiating short commands after they are already in use. Nothing concrete, but I can imagine that that may simplify some implementations. Yeah; I was just thinking of the fact that currently the negotiation is: * send your VERSION message * see what their VERSION message is * announce a bunch of features, depending on the version (or service flags) * send the VERACK (and GETADDR and final ALERT) * wait for their announcements and VERACK * negotiation is finished; we know everything; we're ready to go which only gets you two steps if you send the short id stuff as part of the VERSION message. Obviously you could just add an extra phase either just before or just after the VERACK, though. I suppose being able to choose your own short id mapping from day 0 would mean that every bip324 node could use a single short id mapping for all outgoing messages, which might also make implementation marginally easier (no need to use one table for modern nodes, but also support the original table for old bip324 implementations)... Cheers, aj ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-02-21 16:03 ` Anthony Towns @ 2023-02-28 18:07 ` Dhruv M 2023-02-28 21:02 ` Erik Aronesty 0 siblings, 1 reply; 22+ messages in thread From: Dhruv M @ 2023-02-28 18:07 UTC (permalink / raw) To: bitcoin-dev The relevant changes from this discussion about short 1-byte message type IDs are now in a PR for the bips repo: https://github.com/bitcoin/bips/pull/1428 On 2/21/23 08:03, Anthony Towns via bitcoin-dev wrote: > On Mon, Feb 20, 2023 at 03:22:30PM +0000, Pieter Wuille via bitcoin-dev wrote: >> On Sunday, February 19th, 2023 at 6:56 PM, Anthony Towns <aj@erisian•com.au> wrote: >>> On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via bitcoin-dev wrote: >>>>> I think it's probably less complex to close some of the doors? >>>>> 2) are short ids available/meaningful to send prior to VERACK being >>>>> completed? >>>>> Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for re-negotiating. >>> I think you still need/want two negotiation steps -- once to tell each >>> other what tables you know about, once to choose a mutually recognised >>> table and specify any additions. >> Right, I wasn't talking about how many steps/messages the negotiation takes. I just meant that if all negotiation of the mapping table happens just once (before VERACK) and that negotiation itself happens without use of short commands, then there is no need for re-negotiating short commands after they are already in use. Nothing concrete, but I can imagine that that may simplify some implementations. > Yeah; I was just thinking of the fact that currently the negotiation is: > > * send your VERSION message > * see what their VERSION message is > > * announce a bunch of features, depending on the version (or service > flags) > * send the VERACK (and GETADDR and final ALERT) > > * wait for their announcements and VERACK > * negotiation is finished; we know everything; we're ready to go > > which only gets you two steps if you send the short id stuff as part of > the VERSION message. Obviously you could just add an extra phase either > just before or just after the VERACK, though. > > I suppose being able to choose your own short id mapping from day 0 would > mean that every bip324 node could use a single short id mapping for all > outgoing messages, which might also make implementation marginally easier > (no need to use one table for modern nodes, but also support the original > table for old bip324 implementations)... > > Cheers, > aj > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2023-02-28 18:07 ` Dhruv M @ 2023-02-28 21:02 ` Erik Aronesty 0 siblings, 0 replies; 22+ messages in thread From: Erik Aronesty @ 2023-02-28 21:02 UTC (permalink / raw) To: Dhruv M, Bitcoin Protocol Discussion [-- Attachment #1: Type: text/plain, Size: 3314 bytes --] you can always do what protocols usually do. 1 byte is fine for now, but reserve the top bit for "this is a two byte id" (128 choices). then when you run out of room, set the top bit which means "this is a 2 byte id (again with one reserved) and so-on. ie: how protobuf stores integers. On Tue, Feb 28, 2023 at 1:42 PM Dhruv M via bitcoin-dev < bitcoin-dev@lists•linuxfoundation.org> wrote: > The relevant changes from this discussion about short 1-byte message > type IDs are now in a PR for the bips repo: > https://github.com/bitcoin/bips/pull/1428 > > On 2/21/23 08:03, Anthony Towns via bitcoin-dev wrote: > > On Mon, Feb 20, 2023 at 03:22:30PM +0000, Pieter Wuille via bitcoin-dev > wrote: > >> On Sunday, February 19th, 2023 at 6:56 PM, Anthony Towns < > aj@erisian•com.au> wrote: > >>> On Fri, Feb 17, 2023 at 10:13:05PM +0000, Pieter Wuille via > bitcoin-dev wrote: > >>>>> I think it's probably less complex to close some of the doors? > >>>>> 2) are short ids available/meaningful to send prior to VERACK being > >>>>> completed? > >>>>> Ah, I hadn't considered this nuance. If we don't care about them > being available before VERACK negotiation, then it may be possible to > introduce a way to negotiate a different short id mapping table without > needing a mechanism for re-negotiating. > >>> I think you still need/want two negotiation steps -- once to tell each > >>> other what tables you know about, once to choose a mutually recognised > >>> table and specify any additions. > >> Right, I wasn't talking about how many steps/messages the negotiation > takes. I just meant that if all negotiation of the mapping table happens > just once (before VERACK) and that negotiation itself happens without use > of short commands, then there is no need for re-negotiating short commands > after they are already in use. Nothing concrete, but I can imagine that > that may simplify some implementations. > > Yeah; I was just thinking of the fact that currently the negotiation is: > > > > * send your VERSION message > > * see what their VERSION message is > > > > * announce a bunch of features, depending on the version (or service > > flags) > > * send the VERACK (and GETADDR and final ALERT) > > > > * wait for their announcements and VERACK > > * negotiation is finished; we know everything; we're ready to go > > > > which only gets you two steps if you send the short id stuff as part of > > the VERSION message. Obviously you could just add an extra phase either > > just before or just after the VERACK, though. > > > > I suppose being able to choose your own short id mapping from day 0 would > > mean that every bip324 node could use a single short id mapping for all > > outgoing messages, which might also make implementation marginally easier > > (no need to use one table for modern nodes, but also support the original > > table for old bip324 implementations)... > > > > Cheers, > > aj > > _______________________________________________ > > bitcoin-dev mailing list > > bitcoin-dev@lists•linuxfoundation.org > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > [-- Attachment #2: Type: text/html, Size: 4507 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [bitcoin-dev] Refreshed BIP324 2022-10-08 12:59 [bitcoin-dev] Refreshed BIP324 Dhruv M 2022-10-26 16:39 ` Pieter Wuille @ 2023-10-11 20:52 ` Tim Ruffing 1 sibling, 0 replies; 22+ messages in thread From: Tim Ruffing @ 2023-10-11 20:52 UTC (permalink / raw) To: bitcoin-dev; +Cc: Dhruv M Hello, We'd like to announce two recent updates to BIP324 ("Version 2 P2P Encrypted Transport Protocol"). Some of these changes affect semantics and some are backwards-incompatible. While we are not aware of any implementations of BIP324 except the one in Bitcoin Core (see https://github.com/bitcoin/bitcoin/issues/27634 ), the purpose of the email is to inform anyone involved in other implementation efforts. At this point, we don't expect any further backwards-incompatible changes. https://github.com/bitcoin/bips/pull/1496 did multiple small changes: * Incoming v1 connections are now detected based on first 16 bytes they sent (instead of 12), which improves accuracy. If the incoming v1 connection appears to come from a wrong network (due to non- matching "network magic" bytes), responders may now drop the connection immediately. * The BIP330 message types have been dropped from the short encodings list in the BIP. It feels like it shouldn't be BIP324's goal to predict future protocol improvements. https://github.com/bitcoin/bips/pull/1498 introduced a backwards- incompatible change: * The garbage authentication packet is removed by merging it with the version packet. This simplifies the protocol implementation by consolidating the states and removing the special case of "ignoring the ignore bit." The freedom to choose the contents of the garbage authentication packet has also been removed, leading to easier testing and implementation. We also did some editorial improvements. The most recent revision of the BIP324 can be found at: https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki Best, Dhruv, Tim, and Pieter On Sat, 2022-10-08 at 12:59 +0000, Dhruv M wrote: > Hi all, > > We have refreshed the proposal for BIP324, a new bitcoin P2P protocol > featuring opportunistic encryption, a mild bandwidth reduction, and > the > ability > to negotiate upgrades before exchanging application messages. We'd > like > to invite community members to review the BIP[1] and the related > Bitcoin > Core > code[2]. > > The proposal has a rich history[3]. The big changes since the last > public > appearance[4] are: > > * Elligator-swift encoding for the pubkeys in the ECDH exchange to > obtain a pseudorandom bytestream > * x-only ECDH secret derivation > * Transport versioning that allows for upgradability > * Trafic shapability using decoy packets and a shapable handshake > * Complete rewrite of the BIP text > > We look forward to your review and comments. > > -Dhruv, Tim and Pieter > > > [1] BIP Pull Request: https://github.com/bitcoin/bips/pull/1378 > > [2] All historical and current PRs: > https://bip324.com/sections/code-review/ > > [3] https://bip324.com/sections/bip-review/ > > [4] https://gist.github.com/dhruv/5b1275751bc98f3b64bcafce7876b489 > > > ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2023-10-11 20:58 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-10-08 12:59 [bitcoin-dev] Refreshed BIP324 Dhruv M 2022-10-26 16:39 ` Pieter Wuille 2022-10-27 7:28 ` Vasil Dimov 2022-11-03 17:53 ` Murch 2022-11-03 22:26 ` Jonas Schnelli 2022-11-08 3:20 ` Anthony Towns 2022-11-10 21:23 ` Pieter Wuille 2022-11-12 3:23 ` Pieter Wuille 2022-11-12 18:52 ` Yuval Kogman 2022-11-18 8:24 ` Anthony Towns 2023-01-05 22:06 ` Pieter Wuille 2023-01-05 23:12 ` Anthony Towns 2023-01-09 8:11 ` Anthony Towns 2023-02-16 17:43 ` Dhruv M 2023-02-17 15:51 ` Anthony Towns 2023-02-17 22:13 ` Pieter Wuille 2023-02-19 23:56 ` Anthony Towns 2023-02-20 15:22 ` Pieter Wuille 2023-02-21 16:03 ` Anthony Towns 2023-02-28 18:07 ` Dhruv M 2023-02-28 21:02 ` Erik Aronesty 2023-10-11 20:52 ` Tim Ruffing
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox