public inbox for bitcoindev@googlegroups.com
 help / color / mirror / Atom feed

* [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
@ 2020-08-14 19:28 Suhas Daftuar
  2020-08-16 17:24 ` Jeremy
                   ` (3 more replies)
  0 siblings, 4 replies; 30+ messages in thread
From: Suhas Daftuar @ 2020-08-14 19:28 UTC (permalink / raw)
  To: Bitcoin Dev

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

Hi,

Back in February I posted a proposal for WTXID-based transaction relay[1]
(now known as BIP 339), which included a proposal for feature negotiation
to take place prior to the VERACK message being received by each side.  In
my email to this list, I had asked for feedback as to whether that proposal
was problematic, and didn't receive any responses.

Since then, the implementation of BIP 339 has been merged into Bitcoin
Core, though it has not yet been released.

In thinking about the mechanism used there, I thought it would be helpful
to codify in a BIP the idea that Bitcoin network clients should ignore
unknown messages received before a VERACK.  A draft of my proposal is
available here[2].

I presume that software upgrading past protocol version 70016 was already
planning to either implement BIP 339, or ignore the wtxidrelay message
proposed in BIP 339 (if not, then this would create network split concerns
in the future -- so I hope that someone would speak up if this were a
problem).  When we propose future protocol upgrades that would benefit from
feature negotiation at the time of connection, I think it would be nice to
be able to use the same method as proposed in BIP 339, without even needing
to bump the protocol version.  So having an understanding that this is the
standard of how other network clients operate would be helpful.

If, on the other hand, this is problematic for some reason, I look forward
to hearing that as well, so that we can be careful about how we deploy
future p2p changes to avoid disruption.

Thanks,
Suhas Daftuar

[1]
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html

[2]
https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki

[-- Attachment #2: Type: text/html, Size: 2246 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

• * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
    2020-08-14 19:28 [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup Suhas Daftuar
  @ 2020-08-16 17:24 ` Jeremy
    2020-08-16 19:06 ` Eric Voskuil
                     ` (2 subsequent siblings)
    3 siblings, 0 replies; 30+ messages in thread
  From: Jeremy @ 2020-08-16 17:24 UTC (permalink / raw)
    To: Suhas Daftuar, Bitcoin Protocol Discussion
  
  [-- Attachment #1: Type: text/plain, Size: 891 bytes --]
  
  Concept ack!
  
  It might be nice to include a few negotiation utility functions either in
  this bip or at the same time in a separate bip. An example we might want to
  include is a "polite disconnect", whereby a node can register that you
  don't want to connect in the future due to incompatibility.
  
  It also might be nice to standardize some naming convention or negotiation
  message type so that we don't end up with different negotiation systems.
  Then we can also limit the bip so that we're only defining negotiation
  message types as ignorable v.s. some other message type (which can also be
  ignored, but maybe we want to do something else in the future).
  
  This also makes it easier for old (but newer than this bip) nodes to apply
  some generic rules around reporting/rejecting/responding to unknown feature
  negotiation v.s. an untagged message which might be a negotiation or
  something else.
  
  [-- Attachment #2: Type: text/html, Size: 1078 bytes --]
  
  ^ permalink raw reply	[flat|nested] 30+ messages in thread

• * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
    2020-08-14 19:28 [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup Suhas Daftuar
    2020-08-16 17:24 ` Jeremy
  @ 2020-08-16 19:06 ` Eric Voskuil
    2020-08-17 20:40   ` Suhas Daftuar
    2020-08-20 14:13   ` David A. Harding
    2020-08-18 14:59 ` Matt Corallo
    2020-08-21  2:36 ` Anthony Towns
    3 siblings, 2 replies; 30+ messages in thread
  From: Eric Voskuil @ 2020-08-16 19:06 UTC (permalink / raw)
    To: Suhas Daftuar, Bitcoin Protocol Discussion
  
  [-- Attachment #1: Type: text/plain, Size: 2929 bytes --]
  
  A requirement to ignore unknown (invalid) messages is not only a protocol breaking change but poor protocol design. The purpose of version negotiation is to determine the set of valid messages. Changes to version negotiation itself are very problematic.
  
  The only limitation presented by versioning is that the system is sequential. As such, clients that do not wish to implement (or operators who do not wish to enable) them are faced with a problem when wanting to support later features. This is resolvable by making such features optional at the new protocol level. This allows each client to limit its communication to the negotiated protocol, and allows ignoring of known but unsupported/disabled features.
  
  Sorry I missed your earlier post. Been distracted for a while.
  
  e
  
  
  > On Aug 14, 2020, at 12:28, Suhas Daftuar via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
  > 
  > 
  > Hi,
  > 
  > Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a proposal for feature negotiation to take place prior to the VERACK message being received by each side.  In my email to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
  > 
  > Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
  > 
  > In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin network clients should ignore unknown messages received before a VERACK.  A draft of my proposal is available here[2].
  > 
  > I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future -- so I hope that someone would speak up if this were a problem).  When we propose future protocol upgrades that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method as proposed in BIP 339, without even needing to bump the protocol version.  So having an understanding that this is the standard of how other network clients operate would be helpful.
  > 
  > If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be careful about how we deploy future p2p changes to avoid disruption.
  > 
  > Thanks,
  > Suhas Daftuar
  > 
  > [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html
  > 
  > [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki
  > _______________________________________________
  > bitcoin-dev mailing list
  > bitcoin-dev@lists•linuxfoundation.org
  > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
  
  [-- Attachment #2: Type: text/html, Size: 3868 bytes --]
  
  ^ permalink raw reply	[flat|nested] 30+ messages in thread

• • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
      2020-08-16 19:06 ` Eric Voskuil
    @ 2020-08-17 20:40   ` Suhas Daftuar
      2020-08-17 21:21     ` Eric Voskuil
      2020-08-20 14:13   ` David A. Harding
      1 sibling, 1 reply; 30+ messages in thread
    From: Suhas Daftuar @ 2020-08-17 20:40 UTC (permalink / raw)
      To: Eric Voskuil; +Cc: Bitcoin Protocol Discussion
    
    [-- Attachment #1: Type: text/plain, Size: 5088 bytes --]
    
    Hi Eric,
    
    Thanks for your response.  If I understand correctly, you're suggesting
    that in the future we do the same as what was done in BIP 339, of
    accompanying new messages (which are optional) with a protocol version
    bump, so that network clients are never reading unknown messages from a
    peer (and can be free to disconnect a peer for sending an unknown message)?
    
    I think that works fine, so if indeed there will be software that will
    expect things to operate this way then I can withdraw the suggestion I've
    made in this thread.  However I wanted to clarify that this is what you
    suggest, because there is another downside to this approach (beyond the
    sequential nature of sequence numbers that you mention) -- if a software
    implementation misses a proposed new protocol upgrade, and thus fails to
    parse (and ignore) some proposed new message, the result can be a network
    split down the road as incompatible clients get slowly upgraded over time.
    
    I think this coordination cost is something to be concerned about -- for
    instance, the lack of response to my wtxid-relay proposal made me wonder if
    other software would be implementing something to account for the new
    message that proposal introduces (for clients with version >= 70016).  It's
    reasonable for people to be busy and miss things like this, and I think
    it's worth considering whether there's a safer way for us to deploy changes.
    
    That said, I don't think this coordination cost is unbearable, so as long
    as we have a process for making p2p protocol improvements I'm not too
    worried about what mechanism we use.  So if this concern over coordination
    of changes doesn't sway you, I think we can continue to just bump protocol
    version at the same time as deploying new messages, as we have been doing,
    and hope that we don't run into problems down the road.
    
    If I have misunderstood how you think we should be making future protocol
    changes, please let me know.
    
    Thanks,
    Suhas
    
    
    
    On Sun, Aug 16, 2020 at 3:06 PM Eric Voskuil <eric@voskuil•org> wrote:
    
    > A requirement to ignore unknown (invalid) messages is not only a protocol
    > breaking change but poor protocol design. The purpose of version
    > negotiation is to determine the set of valid messages. Changes to version
    > negotiation itself are very problematic.
    >
    > The only limitation presented by versioning is that the system is
    > sequential. As such, clients that do not wish to implement (or operators
    > who do not wish to enable) them are faced with a problem when wanting to
    > support later features. This is resolvable by making such features optional
    > at the new protocol level. This allows each client to limit its
    > communication to the negotiated protocol, and allows ignoring of known but
    > unsupported/disabled features.
    >
    > Sorry I missed your earlier post. Been distracted for a while.
    >
    > e
    >
    >
    > On Aug 14, 2020, at 12:28, Suhas Daftuar via bitcoin-dev <
    > bitcoin-dev@lists•linuxfoundation.org> wrote:
    >
    > 
    > Hi,
    >
    > Back in February I posted a proposal for WTXID-based transaction relay[1]
    > (now known as BIP 339), which included a proposal for feature negotiation
    > to take place prior to the VERACK message being received by each side.  In
    > my email to this list, I had asked for feedback as to whether that proposal
    > was problematic, and didn't receive any responses.
    >
    > Since then, the implementation of BIP 339 has been merged into Bitcoin
    > Core, though it has not yet been released.
    >
    > In thinking about the mechanism used there, I thought it would be helpful
    > to codify in a BIP the idea that Bitcoin network clients should ignore
    > unknown messages received before a VERACK.  A draft of my proposal is
    > available here[2].
    >
    > I presume that software upgrading past protocol version 70016 was already
    > planning to either implement BIP 339, or ignore the wtxidrelay message
    > proposed in BIP 339 (if not, then this would create network split concerns
    > in the future -- so I hope that someone would speak up if this were a
    > problem).  When we propose future protocol upgrades that would benefit from
    > feature negotiation at the time of connection, I think it would be nice to
    > be able to use the same method as proposed in BIP 339, without even needing
    > to bump the protocol version.  So having an understanding that this is the
    > standard of how other network clients operate would be helpful.
    >
    > If, on the other hand, this is problematic for some reason, I look forward
    > to hearing that as well, so that we can be careful about how we deploy
    > future p2p changes to avoid disruption.
    >
    > Thanks,
    > Suhas Daftuar
    >
    > [1]
    > https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html
    >
    > [2]
    > https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki
    > _______________________________________________
    > bitcoin-dev mailing list
    > bitcoin-dev@lists•linuxfoundation.org
    > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
    >
    >
    
    [-- Attachment #2: Type: text/html, Size: 6560 bytes --]
    
    ^ permalink raw reply	[flat|nested] 30+ messages in thread

  • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
        2020-08-17 20:40   ` Suhas Daftuar
      @ 2020-08-17 21:21     ` Eric Voskuil
        0 siblings, 0 replies; 30+ messages in thread
      From: Eric Voskuil @ 2020-08-17 21:21 UTC (permalink / raw)
        To: Suhas Daftuar; +Cc: Bitcoin Protocol Discussion
      
      [-- Attachment #1: Type: text/plain, Size: 6269 bytes --]
      
      Hi Suhas,
      
      It seems to me that your first two paragraphs contradict each other, so I’m not sure we have understanding. As you say in the first paragraph, a peer would never get messages that it does not understand, so there is no chance that missing a protocol change would matter.
      
      In case it’s not understood, version negotiation provides each peer with the maximum supported protocol version of the other. Once complete both have negotiated to the highest common version. No message not supported at that version may be sent by either.
      
      If the protocol was to accept *any* message traffic then it will cease to be a protocol. People will drop in their changes without obtaining broad support, and peers will evolve to the point of no longer being peers. They won’t speak the same language and the “network” will be little more than a broadcast transport, broadcasting all traffic to all peers, whether it’s for them or not.
      
      People need to either build support or build a distinct network. That’s the actual coordination issue, which is inherent to protocol development.
      
      Best,
      e
      
      > On Aug 17, 2020, at 13:40, Suhas Daftuar <sdaftuar@gmail•com> wrote:
      > 
      > Hi Eric,
      > 
      > Thanks for your response.  If I understand correctly, you're suggesting that in the future we do the same as what was done in BIP 339, of accompanying new messages (which are optional) with a protocol version bump, so that network clients are never reading unknown messages from a peer (and can be free to disconnect a peer for sending an unknown message)?
      > 
      > I think that works fine, so if indeed there will be software that will expect things to operate this way then I can withdraw the suggestion I've made in this thread.  However I wanted to clarify that this is what you suggest, because there is another downside to this approach (beyond the sequential nature of sequence numbers that you mention) -- if a software implementation misses a proposed new protocol upgrade, and thus fails to parse (and ignore) some proposed new message, the result can be a network split down the road as incompatible clients get slowly upgraded over time. 
      > 
      > I think this coordination cost is something to be concerned about -- for instance, the lack of response to my wtxid-relay proposal made me wonder if other software would be implementing something to account for the new message that proposal introduces (for clients with version >= 70016).  It's reasonable for people to be busy and miss things like this, and I think it's worth considering whether there's a safer way for us to deploy changes.
      > 
      > That said, I don't think this coordination cost is unbearable, so as long as we have a process for making p2p protocol improvements I'm not too worried about what mechanism we use.  So if this concern over coordination of changes doesn't sway you, I think we can continue to just bump protocol version at the same time as deploying new messages, as we have been doing, and hope that we don't run into problems down the road.  
      > 
      > If I have misunderstood how you think we should be making future protocol changes, please let me know.
      > 
      > Thanks,
      > Suhas
      > 
      > 
      > 
      >> On Sun, Aug 16, 2020 at 3:06 PM Eric Voskuil <eric@voskuil•org> wrote:
      >> A requirement to ignore unknown (invalid) messages is not only a protocol breaking change but poor protocol design. The purpose of version negotiation is to determine the set of valid messages. Changes to version negotiation itself are very problematic.
      >> 
      >> The only limitation presented by versioning is that the system is sequential. As such, clients that do not wish to implement (or operators who do not wish to enable) them are faced with a problem when wanting to support later features. This is resolvable by making such features optional at the new protocol level. This allows each client to limit its communication to the negotiated protocol, and allows ignoring of known but unsupported/disabled features.
      >> 
      >> Sorry I missed your earlier post. Been distracted for a while.
      >> 
      >> e
      >> 
      >> 
      >>>> On Aug 14, 2020, at 12:28, Suhas Daftuar via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
      >>>> 
      >>> 
      >>> Hi,
      >>> 
      >>> Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a proposal for feature negotiation to take place prior to the VERACK message being received by each side.  In my email to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
      >>> 
      >>> Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
      >>> 
      >>> In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin network clients should ignore unknown messages received before a VERACK.  A draft of my proposal is available here[2].
      >>> 
      >>> I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future -- so I hope that someone would speak up if this were a problem).  When we propose future protocol upgrades that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method as proposed in BIP 339, without even needing to bump the protocol version.  So having an understanding that this is the standard of how other network clients operate would be helpful.
      >>> 
      >>> If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be careful about how we deploy future p2p changes to avoid disruption.
      >>> 
      >>> Thanks,
      >>> Suhas Daftuar
      >>> 
      >>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html
      >>> 
      >>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki
      >>> _______________________________________________
      >>> bitcoin-dev mailing list
      >>> bitcoin-dev@lists•linuxfoundation.org
      >>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
      
      [-- Attachment #2: Type: text/html, Size: 8256 bytes --]
      
      ^ permalink raw reply	[flat|nested] 30+ messages in thread

  • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
      2020-08-16 19:06 ` Eric Voskuil
      2020-08-17 20:40   ` Suhas Daftuar
    @ 2020-08-20 14:13   ` David A. Harding
      1 sibling, 0 replies; 30+ messages in thread
    From: David A. Harding @ 2020-08-20 14:13 UTC (permalink / raw)
      To: Eric Voskuil, Bitcoin Protocol Discussion
    
    [-- Attachment #1: Type: text/plain, Size: 3602 bytes --]
    
    On Sun, Aug 16, 2020 at 12:06:55PM -0700, Eric Voskuil via bitcoin-dev wrote:
    > A requirement to ignore unknown (invalid) messages is [...] a protocol
    > breaking change 
    
    I don't think it is.  The proposed BIP, as currently written, only tells
    nodes to ignore unknown messages during peer negotiation.  The only case
    where this will happen so far is BIP339, which says:
    
        The wtxidrelay message must be sent in response to a VERSION message
        from a peer whose protocol version is >= 70016, and prior to sending
        a VERACK
    
    So unless you signal support for version >=70016, you'll never receive an
    unknown message.  (And, if you do signal, you probably can't claim that
    you were unaware of this new requirement, unless you were using a
    non-BIP protocol like xthin[1]).
    
    However, perhaps this new proposed BIP could be a bit clearer about its
    expectations for future protocol upgrades by saying something like:
    
        Nodes implementing this BIP MUST also not send new negotiation
        message types to nodes whose protocol version is less than 70017.
    
    That should promote backwards compatibility.  If you don't want to
    ignore unknown negotiation messages between `version` and `verack`, you
    can just set your protocol version to a max of 70016.
    
    > A requirement to ignore unknown (invalid) messages is [...] poor
    > protocol design. The purpose of version negotiation is to determine
    > the set of valid messages. 
    
    To be clear, the proposed requirement to ignore unknown messages is
    limited in scope to the brief negotiation phase between `version` and
    `verack`.  If you want to terminate connections (or do whatever) on
    receipt of an unknown message, you can do that at any other time.
    
    > Changes to version negotiation itself are very problematic.
    
    For whom?
    
    > The only limitation presented by versioning is that the system is
    > sequential. 
    
    That seems like a pretty significant limitation to decentralized
    protocol development.
    
    I think there are currently several people who want to run long-term
    experiements for new protocol features using open source opt-in
    codebases that anyone can run, and it would be advantageous to them to
    have a flexible and lightweight feature negotiation system like this
    proposed method.
    
    > As such, clients that do not wish to implement (or operators who do
    > not wish to enable) them are faced with a problem when wanting to
    > support later features. This is resolvable by making such features
    > optional at the new protocol level. This allows each client to limit
    > its communication to the negotiated protocol, and allows ignoring of
    > known but unsupported/disabled features.
    
    I don't understand this.  How do two peers negotiate a set of two or
    more optional features using only the exchange of single numbers?  For
    example:
    
    - Node A supports Feature X (implemented in protocol version 70998) and Feature Y (version 70999).
    
    - Node B does not support X but does want to use Y; what does it use for its
      protocol version number when establishing a connection with node A?
    
    ---
    
    Overall, I like the proposed BIP and the negotiation method it
    describes.
    
    Cheers,
    
    -Dave
    
    [1] This is not a recommendation for xthin, but I do think it's an example
        of the challenges of using a shared linear version number scheme for
        protocol negotiation in a decentralized system where different teams
        don't necessarily get along well with each other.
        https://github.com/ptschip/bitcoinxt/commit/7ea5854a3599851beffb1323544173f03d45373b#diff-c61070c281aed6ded69036c08bd08addR12
    
    [-- Attachment #2: signature.asc --]
    [-- Type: application/pgp-signature, Size: 833 bytes --]
    
    ^ permalink raw reply	[flat|nested] 30+ messages in thread

• * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
    2020-08-14 19:28 [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup Suhas Daftuar
    2020-08-16 17:24 ` Jeremy
    2020-08-16 19:06 ` Eric Voskuil
  @ 2020-08-18 14:59 ` Matt Corallo
    2020-08-18 16:54   ` Eric Voskuil
    2020-08-21  2:36 ` Anthony Towns
    3 siblings, 1 reply; 30+ messages in thread
  From: Matt Corallo @ 2020-08-18 14:59 UTC (permalink / raw)
    To: Suhas Daftuar, Bitcoin Protocol Discussion
  
  This sounds like a great idea!
  
  Bitcoin is no longer a homogeneous network of one client - it is many, with different features implemented in each. The 
  Bitcoin protocol hasn't (fully) evolved to capture that reality. Initially the Bitcoin protocol had a simple numerical 
  version field, but that is wholly impractical for any diverse network - some clients may not wish to implement every 
  possible new relay mechanic, and why should they have to in order to use other new features?
  
  Bitcoin protocol changes have, many times in recent history, been made via new dummy "negotiation" messages, which take 
  advantage of the fact that the Bitcoin protocol has always expected clients to ignore unknown messages. Given that 
  pattern, it makes sense to have an explicit negotiation phase - after version and before verack, just send the list of 
  features that you support to negotiate what the connection will be capable of. The exact way we do that doesn't matter 
  much, and sending it as a stream of messages which each indicate support for a given protocol feature perfectly captures 
  the pattern that has been used in several recent network upgrades, keeping consistency.
  
  Matt
  
  On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
  > Hi,
  > 
  > Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a 
  > proposal for feature negotiation to take place prior to the VERACK message being received by each side.  In my email to 
  > this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
  > 
  > Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
  > 
  > In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin 
  > network clients should ignore unknown messages received before a VERACK.  A draft of my proposal is available here[2].
  > 
  > I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or 
  > ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future 
  > -- so I hope that someone would speak up if this were a problem).  When we propose future protocol upgrades that would 
  > benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method 
  > as proposed in BIP 339, without even needing to bump the protocol version.  So having an understanding that this is the 
  > standard of how other network clients operate would be helpful.
  > 
  > If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be 
  > careful about how we deploy future p2p changes to avoid disruption.
  > 
  > Thanks,
  > Suhas Daftuar
  > 
  > [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html 
  > <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html>
  > 
  > [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki 
  > <https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki>
  > 
  > _______________________________________________
  > bitcoin-dev mailing list
  > bitcoin-dev@lists•linuxfoundation.org
  > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
  > 
  
  
  ^ permalink raw reply	[flat|nested] 30+ messages in thread

• • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
      2020-08-18 14:59 ` Matt Corallo
    @ 2020-08-18 16:54   ` Eric Voskuil
      2020-08-18 17:26     ` Matt Corallo
      0 siblings, 1 reply; 30+ messages in thread
    From: Eric Voskuil @ 2020-08-18 16:54 UTC (permalink / raw)
      To: Matt Corallo, Bitcoin Protocol Discussion
    
    [-- Attachment #1: Type: text/plain, Size: 5201 bytes --]
    
    “Bitcoin protocol has always expected clients to ignore unknown messages”
    
    This is not true. Bitcoin has long implemented version negotiation, which is the opposite expectation. Libbitcoin’s p2p protocol implementation immediately drops a peer that sends an invalid message according to the negotiated version. The fact that a given client does not validate the protocol does not make it an expectation that the protocol not be validated.
    
    Features can clearly be optional within an actual protocol. There have been post-handshake negotiations implemented for optional messages which are valid at the negotiated version. The protocol may be flexible while remaining validateable. There is no reason to force a client to accept unknown message traffic.
    
    A generalized versioning change can be implemented in or after the handshake. The latter is already done on an ad-hoc basis. The former is possible as long as the peer’s version is sufficient to be aware of the behavior. This does not imply any need to send invalid messages. The verack itself can simply be extended with a matrix of feature support. There is no reason to complicate negotiation with an additional message(s).
    
    FWIW, bip37 did this poorly, adding a feature field to the version message, resulting in bip60. Due to this design, older protocol-validating clients were broken. In this case it was message length that was presumed to not be validated.
    
    e
    
    > On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
    > 
    > This sounds like a great idea!
    > 
    > Bitcoin is no longer a homogeneous network of one client - it is many, with different features implemented in each. The Bitcoin protocol hasn't (fully) evolved to capture that reality. Initially the Bitcoin protocol had a simple numerical version field, but that is wholly impractical for any diverse network - some clients may not wish to implement every possible new relay mechanic, and why should they have to in order to use other new features?
    > 
    > Bitcoin protocol changes have, many times in recent history, been made via new dummy "negotiation" messages, which take advantage of the fact that the Bitcoin protocol has always expected clients to ignore unknown messages. Given that pattern, it makes sense to have an explicit negotiation phase - after version and before verack, just send the list of features that you support to negotiate what the connection will be capable of. The exact way we do that doesn't matter much, and sending it as a stream of messages which each indicate support for a given protocol feature perfectly captures the pattern that has been used in several recent network upgrades, keeping consistency.
    > 
    > Matt
    > 
    >> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
    >> Hi,
    >> Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a proposal for feature negotiation to take place prior to the VERACK message being received by each side.  In my email to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
    >> Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
    >> In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin network clients should ignore unknown messages received before a VERACK.  A draft of my proposal is available here[2].
    >> I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future -- so I hope that someone would speak up if this were a problem).  When we propose future protocol upgrades that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method as proposed in BIP 339, without even needing to bump the protocol version.  So having an understanding that this is the standard of how other network clients operate would be helpful.
    >> If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be careful about how we deploy future p2p changes to avoid disruption.
    >> Thanks,
    >> Suhas Daftuar
    >> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html>
    >> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki>
    >> _______________________________________________
    >> bitcoin-dev mailing list
    >> bitcoin-dev@lists•linuxfoundation.org
    >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
    > _______________________________________________
    > bitcoin-dev mailing list
    > bitcoin-dev@lists•linuxfoundation.org
    > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
    
    [-- Attachment #2: Type: text/html, Size: 7084 bytes --]
    
    ^ permalink raw reply	[flat|nested] 30+ messages in thread

  • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
        2020-08-18 16:54   ` Eric Voskuil
      @ 2020-08-18 17:26     ` Matt Corallo
        2020-08-18 18:11       ` Eric Voskuil
        0 siblings, 1 reply; 30+ messages in thread
      From: Matt Corallo @ 2020-08-18 17:26 UTC (permalink / raw)
        To: Eric Voskuil, Bitcoin Protocol Discussion
      
      There are several cases where a new message has been sent as a part of a negotiation without changing the protocol 
      version. You may chose to ignore that, but that doesn't mean that it isn't an understood and even relied upon feature of 
      the Bitcoin P2P protocol. If you wish to fail connections to new nodes (and risk network splits, as Suhas points out), 
      then you may do so, but that doesn't make it a part of the Bitcoin P2P protocol that you must do so. Of course there is 
      no "official document" by which we can make a formal appeal, but historical precedent suggests otherwise.
      
      Still, I think we're talking pedantics here, and not in a useful way. Ultimately we need some kind of negotiation which 
      is flexible in allowing different software to negotiate different features without a global lock-step version number 
      increase. Or, to put it another way, if a feature is fully optional, why should there be a version number increase for 
      it - the negotiation of it is independent and a version number only increases confusion over which change "owns" a given 
      version number.
      
      I presume you'd support a single message that lists the set of features which a node (optionally) wishes to support on 
      the connection. This proposal is fully equivalent to that, instead opting to list them as individual messages instead of 
      one message, which is a bit nicer in that they can be handled more independently or by different subsystems including 
      even the message hashing.
      
      Matt
      
      On 8/18/20 12:54 PM, Eric Voskuil wrote:
      > “Bitcoin protocol has always expected clients to ignore unknown messages”
      > 
      > This is not true. Bitcoin has long implemented version negotiation, which is the opposite expectation. Libbitcoin’s p2p 
      > protocol implementation immediately drops a peer that sends an invalid message according to the negotiated version. The 
      > fact that a given client does not validate the protocol does not make it an expectation that the protocol not be validated.
      > 
      > Features can clearly be optional within an actual protocol. There have been post-handshake negotiations implemented for 
      > optional messages which are valid at the negotiated version. The protocol may be flexible while remaining validateable. 
      > There is no reason to force a client to accept unknown message traffic.
      > 
      > A generalized versioning change can be implemented in or after the handshake. The latter is already done on an ad-hoc 
      > basis. The former is possible as long as the peer’s version is sufficient to be aware of the behavior. This does not 
      > imply any need to send invalid messages. The verack itself can simply be extended with a matrix of feature support. 
      > There is no reason to complicate negotiation with an additional message(s).
      > 
      > FWIW, bip37 did this poorly, adding a feature field to the version message, resulting in bip60. Due to this design, 
      > older protocol-validating clients were broken. In this case it was message length that was presumed to not be validated.
      > 
      > e
      > 
      >> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
      >>
      >> This sounds like a great idea!
      >>
      >> Bitcoin is no longer a homogeneous network of one client - it is many, with different features implemented in each. 
      >> The Bitcoin protocol hasn't (fully) evolved to capture that reality. Initially the Bitcoin protocol had a simple 
      >> numerical version field, but that is wholly impractical for any diverse network - some clients may not wish to 
      >> implement every possible new relay mechanic, and why should they have to in order to use other new features?
      >>
      >> Bitcoin protocol changes have, many times in recent history, been made via new dummy "negotiation" messages, which 
      >> take advantage of the fact that the Bitcoin protocol has always expected clients to ignore unknown messages. Given 
      >> that pattern, it makes sense to have an explicit negotiation phase - after version and before verack, just send the 
      >> list of features that you support to negotiate what the connection will be capable of. The exact way we do that 
      >> doesn't matter much, and sending it as a stream of messages which each indicate support for a given protocol feature 
      >> perfectly captures the pattern that has been used in several recent network upgrades, keeping consistency.
      >>
      >> Matt
      >>
      >> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
      >>> Hi,
      >>> Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a 
      >>> proposal for feature negotiation to take place prior to the VERACK message being received by each side.  In my email 
      >>> to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
      >>> Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
      >>> In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin 
      >>> network clients should ignore unknown messages received before a VERACK.  A draft of my proposal is available here[2].
      >>> I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or 
      >>> ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the 
      >>> future -- so I hope that someone would speak up if this were a problem).  When we propose future protocol upgrades 
      >>> that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the 
      >>> same method as proposed in BIP 339, without even needing to bump the protocol version.  So having an understanding 
      >>> that this is the standard of how other network clients operate would be helpful.
      >>> If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be 
      >>> careful about how we deploy future p2p changes to avoid disruption.
      >>> Thanks,
      >>> Suhas Daftuar
      >>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html 
      >>> <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html>
      >>> [2] 
      >>> https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki 
      >>> <https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki>
      >>> _______________________________________________
      >>> bitcoin-dev mailing list
      >>> bitcoin-dev@lists•linuxfoundation.org
      >>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
      >> _______________________________________________
      >> bitcoin-dev mailing list
      >> bitcoin-dev@lists•linuxfoundation.org
      >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
      
      
      ^ permalink raw reply	[flat|nested] 30+ messages in thread

    • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
          2020-08-18 17:26     ` Matt Corallo
        @ 2020-08-18 18:11       ` Eric Voskuil
          2020-08-18 18:25         ` Matt Corallo
          0 siblings, 1 reply; 30+ messages in thread
        From: Eric Voskuil @ 2020-08-18 18:11 UTC (permalink / raw)
          To: Matt Corallo; +Cc: Bitcoin Protocol Discussion
        
        
        > On Aug 18, 2020, at 10:26, Matt Corallo <lf-lists@mattcorallo•com> wrote:
        > 
        > There are several cases where a new message has been sent as a part of a negotiation without changing the protocol version.
        
        Such as?
        
        > You may chose to ignore that, but that doesn't mean that it isn't an understood and even relied upon feature of the Bitcoin P2P protocol. If you wish to fail connections to new nodes (and risk network splits, as Suhas points out), then you may do so, but that doesn't make it a part of the Bitcoin P2P protocol that you must do so. Of course there is no "official document" by which we can make a formal appeal, but historical precedent suggests otherwise.
        
        You are misrepresenting “historical precedent”. I’ve seen several attempts to require arbitrary traffic over the years and none have been realized.
        
        > Still, I think we're talking pedantics here, and not in a useful way.
        
        Not to be pedantic, but I don’t know what that means.
        
        > Ultimately we need some kind of negotiation which is flexible in allowing different software to negotiate different features without a global lock-step version number increase.
        
        I have shown below how that already works.
        
        > Or, to put it another way, if a feature is fully optional, why should there be a version number increase for it
        
        For the reasons previously given.
        
        > - the negotiation of it is independent and a version number only increases confusion over which change "owns" a given version number.
        
        Presumably this is why we have a standards process. Any new message implies ownership. Deconflicting that is required, which implies it can easily be version associated (as it has been).
        
        > I presume you'd support a single message that lists the set of features which a node (optionally) wishes to support on the connection. This proposal is fully equivalent to that, instead opting to list them as individual messages instead of one message, which is a bit nicer in that they can be handled more independently or by different subsystems including even the message hashing.
        
        This presumes an implementation. As part of the handshake, collection of an arbitrary set of messages is a significant and unnecessary complication *of the protocol*. Extension of the verack is not. It is the simplest change possible to implement the desired behavior. Each peer simply supplies the matrix of sub-protocols it supports and only those supported by both are allowed. There is no reason for the protocol to split that matrix up into multiple messages, requiring termination. Independent messages exist because of timing or ordering requirements. Implementing dependent messages as if they were independent is wasteful and complicating.
        
        I’m well aware of the inefficiency produced by version linearity in the face of optional sub-protocols. The protocol must negotiate to the version where it can then negotiate support, which has been done. I support creating a simpler system, eliminating these extra messages. The existing numeric version can be reserved exclusively for “must” implement, and can be used to signal an extension to the verack. The verack can then carry a list of “may” or “should” sub-protocols for final negotiation.
        
        The format of the matrix is arbitrary, but the requirement is to list a set of optional sub-protocols. This implies a namespace. This implies “ownership“ of names. In other words, that coordination requirement is not eliminated.
        
        e
        
        > Matt
        > 
        >> On 8/18/20 12:54 PM, Eric Voskuil wrote:
        >> “Bitcoin protocol has always expected clients to ignore unknown messages”
        >> This is not true. Bitcoin has long implemented version negotiation, which is the opposite expectation. Libbitcoin’s p2p protocol implementation immediately drops a peer that sends an invalid message according to the negotiated version. The fact that a given client does not validate the protocol does not make it an expectation that the protocol not be validated.
        >> Features can clearly be optional within an actual protocol. There have been post-handshake negotiations implemented for optional messages which are valid at the negotiated version. The protocol may be flexible while remaining validateable. There is no reason to force a client to accept unknown message traffic.
        >> A generalized versioning change can be implemented in or after the handshake. The latter is already done on an ad-hoc basis. The former is possible as long as the peer’s version is sufficient to be aware of the behavior. This does not imply any need to send invalid messages. The verack itself can simply be extended with a matrix of feature support. There is no reason to complicate negotiation with an additional message(s).
        >> FWIW, bip37 did this poorly, adding a feature field to the version message, resulting in bip60. Due to this design, older protocol-validating clients were broken. In this case it was message length that was presumed to not be validated.
        >> e
        >>>> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
        >>> 
        >>> This sounds like a great idea!
        >>> 
        >>> Bitcoin is no longer a homogeneous network of one client - it is many, with different features implemented in each. The Bitcoin protocol hasn't (fully) evolved to capture that reality. Initially the Bitcoin protocol had a simple numerical version field, but that is wholly impractical for any diverse network - some clients may not wish to implement every possible new relay mechanic, and why should they have to in order to use other new features?
        >>> 
        >>> Bitcoin protocol changes have, many times in recent history, been made via new dummy "negotiation" messages, which take advantage of the fact that the Bitcoin protocol has always expected clients to ignore unknown messages. Given that pattern, it makes sense to have an explicit negotiation phase - after version and before verack, just send the list of features that you support to negotiate what the connection will be capable of. The exact way we do that doesn't matter much, and sending it as a stream of messages which each indicate support for a given protocol feature perfectly captures the pattern that has been used in several recent network upgrades, keeping consistency.
        >>> 
        >>> Matt
        >>> 
        >>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
        >>>> Hi,
        >>>> Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a proposal for feature negotiation to take place prior to the VERACK message being received by each side.  In my email to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
        >>>> Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
        >>>> In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin network clients should ignore unknown messages received before a VERACK.  A draft of my proposal is available here[2].
        >>>> I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future -- so I hope that someone would speak up if this were a problem).  When we propose future protocol upgrades that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method as proposed in BIP 339, without even needing to bump the protocol version.  So having an understanding that this is the standard of how other network clients operate would be helpful.
        >>>> If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be careful about how we deploy future p2p changes to avoid disruption.
        >>>> Thanks,
        >>>> Suhas Daftuar
        >>>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html>
        >>>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki>
        >>>> _______________________________________________
        >>>> bitcoin-dev mailing list
        >>>> bitcoin-dev@lists•linuxfoundation.org
        >>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
        >>> _______________________________________________
        >>> bitcoin-dev mailing list
        >>> bitcoin-dev@lists•linuxfoundation.org
        >>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
        
        
        ^ permalink raw reply	[flat|nested] 30+ messages in thread

      • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
            2020-08-18 18:11       ` Eric Voskuil
          @ 2020-08-18 18:25         ` Matt Corallo
            2020-08-18 18:56           ` Eric Voskuil
            0 siblings, 1 reply; 30+ messages in thread
          From: Matt Corallo @ 2020-08-18 18:25 UTC (permalink / raw)
            To: Eric Voskuil; +Cc: Bitcoin Protocol Discussion
          
          
          
          On 8/18/20 2:11 PM, Eric Voskuil wrote:
            - snip -
          >> Still, I think we're talking pedantics here, and not in a useful way.
          > 
          > Not to be pedantic, but I don’t know what that means.
          
          It means that part of the discussion is not useful, and not worth bothering to go back and figure out what was shipped 
          before the version increase and what wasn't, lets talk about what makes sense for the future :).
          
          >> Ultimately we need some kind of negotiation which is flexible in allowing different software to negotiate different features without a global lock-step version number increase.
          > 
          > I have shown below how that already works.
          > 
          >> Or, to put it another way, if a feature is fully optional, why should there be a version number increase for it
          > 
          > For the reasons previously given.
          > 
          >> - the negotiation of it is independent and a version number only increases confusion over which change "owns" a given version number.
          > 
          > Presumably this is why we have a standards process. Any new message implies ownership. Deconflicting that is required, which implies it can easily be version associated (as it has been).
          
          I think the point is, this doesn't work today, bumping the protocol version requires everyone agreeing on which features 
          make sense, and as we can see from this email thread alone, that isn't a common result in this community. People happily 
          ignore BIPs that make no sense, of which there are a lot, and they should totally be able to do that!
          
          You can say that the current world works, but there's a reason over time we've shifted away from the original "shove 
          another bit on the end of the version message, and everyone agrees on the order of those bits for new feature 
          negotiation." Version bumping is an extension of that, really.
          
          >> I presume you'd support a single message that lists the set of features which a node (optionally) wishes to support on the connection. This proposal is fully equivalent to that, instead opting to list them as individual messages instead of one message, which is a bit nicer in that they can be handled more independently or by different subsystems including even the message hashing.
          > 
          > This presumes an implementation. As part of the handshake, collection of an arbitrary set of messages is a significant and unnecessary complication *of the protocol*. Extension of the verack is not. It is the simplest change possible to implement the desired behavior. Each peer simply supplies the matrix of sub-protocols it supports and only those supported by both are allowed. There is no reason for the protocol to split that matrix up into multiple messages, requiring termination. Independent messages exist because of timing or ordering requirements. Implementing dependent messages as if they were independent is wasteful and complicating.
          
          Some things may need further negotiation. eg compact blocks sends multiple redundant messages with different versions 
          and then deduces the correct version based on the message ordering and version set supported. Doing this via verack 
          locks you into a very specific possible negotiation protocols. You could extend it further and suggest a verack K-V list 
          which allows for more flexible negotiation, but I'm not sure that it isn't more complicated than just shoving more 
          messages on the wire.
          
          > I’m well aware of the inefficiency produced by version linearity in the face of optional sub-protocols. The protocol must negotiate to the version where it can then negotiate support, which has been done. I support creating a simpler system, eliminating these extra messages. The existing numeric version can be reserved exclusively for “must” implement, and can be used to signal an extension to the verack. The verack can then carry a list of “may” or “should” sub-protocols for final negotiation.
          
          I think we agree here - the current method of protocol version bumping isn't scalable and something more flexible is 
          definitely a better world.
          
          > The format of the matrix is arbitrary, but the requirement is to list a set of optional sub-protocols. This implies a namespace. This implies “ownership“ of names. In other words, that coordination requirement is not eliminated.
          
          This is true, there is some ownership requirement, we could switch to hashes or something of the like, but 
          human-readable names have shown to be relatively non-colliding in past Bitcoin protocol changes.
          
          > e
          > 
          >> Matt
          >>
          >>> On 8/18/20 12:54 PM, Eric Voskuil wrote:
          >>> “Bitcoin protocol has always expected clients to ignore unknown messages”
          >>> This is not true. Bitcoin has long implemented version negotiation, which is the opposite expectation. Libbitcoin’s p2p protocol implementation immediately drops a peer that sends an invalid message according to the negotiated version. The fact that a given client does not validate the protocol does not make it an expectation that the protocol not be validated.
          >>> Features can clearly be optional within an actual protocol. There have been post-handshake negotiations implemented for optional messages which are valid at the negotiated version. The protocol may be flexible while remaining validateable. There is no reason to force a client to accept unknown message traffic.
          >>> A generalized versioning change can be implemented in or after the handshake. The latter is already done on an ad-hoc basis. The former is possible as long as the peer’s version is sufficient to be aware of the behavior. This does not imply any need to send invalid messages. The verack itself can simply be extended with a matrix of feature support. There is no reason to complicate negotiation with an additional message(s).
          >>> FWIW, bip37 did this poorly, adding a feature field to the version message, resulting in bip60. Due to this design, older protocol-validating clients were broken. In this case it was message length that was presumed to not be validated.
          >>> e
          >>>>> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
          >>>>
          >>>> This sounds like a great idea!
          >>>>
          >>>> Bitcoin is no longer a homogeneous network of one client - it is many, with different features implemented in each. The Bitcoin protocol hasn't (fully) evolved to capture that reality. Initially the Bitcoin protocol had a simple numerical version field, but that is wholly impractical for any diverse network - some clients may not wish to implement every possible new relay mechanic, and why should they have to in order to use other new features?
          >>>>
          >>>> Bitcoin protocol changes have, many times in recent history, been made via new dummy "negotiation" messages, which take advantage of the fact that the Bitcoin protocol has always expected clients to ignore unknown messages. Given that pattern, it makes sense to have an explicit negotiation phase - after version and before verack, just send the list of features that you support to negotiate what the connection will be capable of. The exact way we do that doesn't matter much, and sending it as a stream of messages which each indicate support for a given protocol feature perfectly captures the pattern that has been used in several recent network upgrades, keeping consistency.
          >>>>
          >>>> Matt
          >>>>
          >>>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
          >>>>> Hi,
          >>>>> Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a proposal for feature negotiation to take place prior to the VERACK message being received by each side.  In my email to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
          >>>>> Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
          >>>>> In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin network clients should ignore unknown messages received before a VERACK.  A draft of my proposal is available here[2].
          >>>>> I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future -- so I hope that someone would speak up if this were a problem).  When we propose future protocol upgrades that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method as proposed in BIP 339, without even needing to bump the protocol version.  So having an understanding that this is the standard of how other network clients operate would be helpful.
          >>>>> If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be careful about how we deploy future p2p changes to avoid disruption.
          >>>>> Thanks,
          >>>>> Suhas Daftuar
          >>>>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html>
          >>>>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki>
          >>>>> _______________________________________________
          >>>>> bitcoin-dev mailing list
          >>>>> bitcoin-dev@lists•linuxfoundation.org
          >>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
          >>>> _______________________________________________
          >>>> bitcoin-dev mailing list
          >>>> bitcoin-dev@lists•linuxfoundation.org
          >>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
          
          
          ^ permalink raw reply	[flat|nested] 30+ messages in thread

        • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
              2020-08-18 18:25         ` Matt Corallo
            @ 2020-08-18 18:56           ` Eric Voskuil
              0 siblings, 0 replies; 30+ messages in thread
            From: Eric Voskuil @ 2020-08-18 18:56 UTC (permalink / raw)
              To: Matt Corallo; +Cc: Bitcoin Protocol Discussion
            
            
            > On Aug 18, 2020, at 11:25, Matt Corallo <lf-lists@mattcorallo•com> wrote:
            > 
            > On 8/18/20 2:11 PM, Eric Voskuil wrote:
            > - snip -
            >>> Still, I think we're talking pedantics here, and not in a useful way.
            >> Not to be pedantic, but I don’t know what that means.
            > 
            > It means that part of the discussion is not useful, and not worth bothering to go back and figure out what was shipped before the version increase and what wasn't, lets talk about what makes sense for the future :).
            
            When the discussion centers on backward compatibility, and there is confusion over what that actually implies, this is a central question. You snipped the bit about what actually constitutes the existing protocol. I have implemented every aspect of the protocol that is widely deployed, and I can say without question that the protocol does not require a peer to accept arbitrary messages. In other words, your statement on the subject was a very relevant factual error. Furthermore no reason has been demonstrated here to accept arbitrary traffic.
            
            >>> Ultimately we need some kind of negotiation which is flexible in allowing different software to negotiate different features without a global lock-step version number increase.
            >> I have shown below how that already works.
            >>> Or, to put it another way, if a feature is fully optional, why should there be a version number increase for it
            >> For the reasons previously given.
            >>> - the negotiation of it is independent and a version number only increases confusion over which change "owns" a given version number.
            >> Presumably this is why we have a standards process. Any new message implies ownership. Deconflicting that is required, which implies it can easily be version associated (as it has been).
            > 
            > I think the point is, this doesn't work today, bumping the protocol version requires everyone agreeing on which features make sense,
            
            As I have shown, this is not the case. While I have given my support to simplifying the process, we should not proceed based on an incorrect understanding of actual behavior.
            
            > and as we can see from this email thread alone, that isn't a common result in this community. People happily ignore BIPs that make no sense, of which there are a lot, and they should totally be able to do that!
            > 
            > You can say that the current world works, but there's a reason over time we've shifted away from the original "shove another bit on the end of the version message, and everyone agrees on the order of those bits for new feature negotiation." Version bumping is an extension of that, really.
            
            Actually the protocol has not done this. It has used the version to signal a new sub-protocol, and then in some cases that sub-protocol has been made optional through subsequent negotiation. What is being proposed here is to simplify that process by collapsing the secondary negotiation into the handshake.
            
            In fact I argued against this secondary ad-hoc negotiation when it began. Now we are coming around to recognizing that it’s a handshake issue, as I said at the time. I’m glad to see that.
            
            >>> I presume you'd support a single message that lists the set of features which a node (optionally) wishes to support on the connection. This proposal is fully equivalent to that, instead opting to list them as individual messages instead of one message, which is a bit nicer in that they can be handled more independently or by different subsystems including even the message hashing.
            >> This presumes an implementation. As part of the handshake, collection of an arbitrary set of messages is a significant and unnecessary complication *of the protocol*. Extension of the verack is not. It is the simplest change possible to implement the desired behavior. Each peer simply supplies the matrix of sub-protocols it supports and only those supported by both are allowed. There is no reason for the protocol to split that matrix up into multiple messages, requiring termination. Independent messages exist because of timing or ordering requirements. Implementing dependent messages as if they were independent is wasteful and complicating.
            > 
            > Some things may need further negotiation. eg compact blocks sends multiple redundant messages with different versions and then deduces the correct version based on the message ordering and version set supported. Doing this via verack locks you into a very specific possible negotiation protocols.
            
            This is a moot point. Whether a list of supported optional sub-protocols is listed in one or multiple messages in the handshake would not change this.
            
            > You could extend it further and suggest a verack K-V list which allows for more flexible negotiation, but I'm not sure that it isn't more complicated than just shoving more messages on the wire.
            
            There is no difference between a constrained set of key-value pairs and a distinct set of options, so there is no additional complexity here.
            
            >> I’m well aware of the inefficiency produced by version linearity in the face of optional sub-protocols. The protocol must negotiate to the version where it can then negotiate support, which has been done. I support creating a simpler system, eliminating these extra messages. The existing numeric version can be reserved exclusively for “must” implement, and can be used to signal an extension to the verack. The verack can then carry a list of “may” or “should” sub-protocols for final negotiation.
            > 
            > I think we agree here - the current method of protocol version bumping isn't scalable and something more flexible is definitely a better world.
            
            To be clear this does not increase flexibility, it reduces communication and therefore complexity, and allows peers to lock in allowed message semantics by the end of the handshake, as opposed to allowing them to change at any time.
            
            >> The format of the matrix is arbitrary, but the requirement is to list a set of optional sub-protocols. This implies a namespace. This implies “ownership“ of names. In other words, that coordination requirement is not eliminated.
            > 
            > This is true, there is some ownership requirement, we could switch to hashes or something of the like, but human-readable names have shown to be relatively non-colliding in past Bitcoin protocol changes.
            
            Hashes don’t prevent collisions. Someone can just use the same hash. Bitcoin uses names (message names) and numbers (version, service, relay...). It’s a protocol, coordination is the whole point.
            
            e
            
            >> e
            >>> Matt
            >>> 
            >>>> On 8/18/20 12:54 PM, Eric Voskuil wrote:
            >>>> “Bitcoin protocol has always expected clients to ignore unknown messages”
            >>>> This is not true. Bitcoin has long implemented version negotiation, which is the opposite expectation. Libbitcoin’s p2p protocol implementation immediately drops a peer that sends an invalid message according to the negotiated version. The fact that a given client does not validate the protocol does not make it an expectation that the protocol not be validated.
            >>>> Features can clearly be optional within an actual protocol. There have been post-handshake negotiations implemented for optional messages which are valid at the negotiated version. The protocol may be flexible while remaining validateable. There is no reason to force a client to accept unknown message traffic.
            >>>> A generalized versioning change can be implemented in or after the handshake. The latter is already done on an ad-hoc basis. The former is possible as long as the peer’s version is sufficient to be aware of the behavior. This does not imply any need to send invalid messages. The verack itself can simply be extended with a matrix of feature support. There is no reason to complicate negotiation with an additional message(s).
            >>>> FWIW, bip37 did this poorly, adding a feature field to the version message, resulting in bip60. Due to this design, older protocol-validating clients were broken. In this case it was message length that was presumed to not be validated.
            >>>> e
            >>>>>> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
            >>>>> 
            >>>>> This sounds like a great idea!
            >>>>> 
            >>>>> Bitcoin is no longer a homogeneous network of one client - it is many, with different features implemented in each. The Bitcoin protocol hasn't (fully) evolved to capture that reality. Initially the Bitcoin protocol had a simple numerical version field, but that is wholly impractical for any diverse network - some clients may not wish to implement every possible new relay mechanic, and why should they have to in order to use other new features?
            >>>>> 
            >>>>> Bitcoin protocol changes have, many times in recent history, been made via new dummy "negotiation" messages, which take advantage of the fact that the Bitcoin protocol has always expected clients to ignore unknown messages. Given that pattern, it makes sense to have an explicit negotiation phase - after version and before verack, just send the list of features that you support to negotiate what the connection will be capable of. The exact way we do that doesn't matter much, and sending it as a stream of messages which each indicate support for a given protocol feature perfectly captures the pattern that has been used in several recent network upgrades, keeping consistency.
            >>>>> 
            >>>>> Matt
            >>>>> 
            >>>>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
            >>>>>> Hi,
            >>>>>> Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a proposal for feature negotiation to take place prior to the VERACK message being received by each side.  In my email to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
            >>>>>> Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
            >>>>>> In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin network clients should ignore unknown messages received before a VERACK.  A draft of my proposal is available here[2].
            >>>>>> I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future -- so I hope that someone would speak up if this were a problem).  When we propose future protocol upgrades that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method as proposed in BIP 339, without even needing to bump the protocol version.  So having an understanding that this is the standard of how other network clients operate would be helpful.
            >>>>>> If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be careful about how we deploy future p2p changes to avoid disruption.
            >>>>>> Thanks,
            >>>>>> Suhas Daftuar
            >>>>>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html>
            >>>>>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki>
            >>>>>> _______________________________________________
            >>>>>> bitcoin-dev mailing list
            >>>>>> bitcoin-dev@lists•linuxfoundation.org
            >>>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
            >>>>> _______________________________________________
            >>>>> bitcoin-dev mailing list
            >>>>> bitcoin-dev@lists•linuxfoundation.org
            >>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
            
            
            ^ permalink raw reply	[flat|nested] 30+ messages in thread

• * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
    2020-08-14 19:28 [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup Suhas Daftuar
                     ` (2 preceding siblings ...)
    2020-08-18 14:59 ` Matt Corallo
  @ 2020-08-21  2:36 ` Anthony Towns
    2020-08-21  4:25   ` Eric Voskuil
    2020-08-21 14:15   ` lf-lists
    3 siblings, 2 replies; 30+ messages in thread
  From: Anthony Towns @ 2020-08-21  2:36 UTC (permalink / raw)
    To: Suhas Daftuar, Bitcoin Protocol Discussion
  
  On Fri, Aug 14, 2020 at 03:28:41PM -0400, Suhas Daftuar via bitcoin-dev wrote:
  > In thinking about the mechanism used there, I thought it would be helpful to
  > codify in a BIP the idea that Bitcoin network clients should ignore unknown
  > messages received before a VERACK.  A draft of my proposal is available here
  > [2].
  
  Rather than allowing arbitrary messages, maybe it would make sense to
  have a specific feature negotiation message, eg:
  
    VERSION ...
    FEATURE wtxidrelay
    FEATURE packagerelay
    VERACK
  
  with the behaviour being that it's valid only between VERSION and VERACK,
  and it takes a length-prefixed-string giving the feature name, optional
  additional data, and if the feature name isn't recognised the message
  is ignored.
  
  If we were to support a "polite disconnect" feature like Jeremy suggested,
  it might be easier to do that for a generic FEATURE message, than
  reimplement it for the message proposed by each new feature.
  
  Cheers,
  aj
  
  
  
  ^ permalink raw reply	[flat|nested] 30+ messages in thread

• • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
      2020-08-21  2:36 ` Anthony Towns
    @ 2020-08-21  4:25   ` Eric Voskuil
      2020-08-21 14:15   ` lf-lists
      1 sibling, 0 replies; 30+ messages in thread
    From: Eric Voskuil @ 2020-08-21  4:25 UTC (permalink / raw)
      To: Anthony Towns, Bitcoin Protocol Discussion
    
    Hi Anthony,
    
    This is what I was implying in my last post (the reference to the unnecessary overload of message typing). However, if one imagines a sequence diagram for this communication it becomes obvious that all such messages are 100% redundant with verack.
    
    e
    
    > On Aug 20, 2020, at 19:37, Anthony Towns via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
    > 
    > On Fri, Aug 14, 2020 at 03:28:41PM -0400, Suhas Daftuar via bitcoin-dev wrote:
    >> In thinking about the mechanism used there, I thought it would be helpful to
    >> codify in a BIP the idea that Bitcoin network clients should ignore unknown
    >> messages received before a VERACK.  A draft of my proposal is available here
    >> [2].
    > 
    > Rather than allowing arbitrary messages, maybe it would make sense to
    > have a specific feature negotiation message, eg:
    > 
    >  VERSION ...
    >  FEATURE wtxidrelay
    >  FEATURE packagerelay
    >  VERACK
    > 
    > with the behaviour being that it's valid only between VERSION and VERACK,
    > and it takes a length-prefixed-string giving the feature name, optional
    > additional data, and if the feature name isn't recognised the message
    > is ignored.
    > 
    > If we were to support a "polite disconnect" feature like Jeremy suggested,
    > it might be easier to do that for a generic FEATURE message, than
    > reimplement it for the message proposed by each new feature.
    > 
    > Cheers,
    > aj
    > 
    > _______________________________________________
    > bitcoin-dev mailing list
    > bitcoin-dev@lists•linuxfoundation.org
    > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
    
    
    ^ permalink raw reply	[flat|nested] 30+ messages in thread

  • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
      2020-08-21  2:36 ` Anthony Towns
      2020-08-21  4:25   ` Eric Voskuil
    @ 2020-08-21 14:15   ` lf-lists
      2020-08-21 16:42     ` Eric Voskuil
      1 sibling, 1 reply; 30+ messages in thread
    From: lf-lists @ 2020-08-21 14:15 UTC (permalink / raw)
      To: Anthony Towns, Bitcoin Protocol Discussion
    
    Sure, we could do a new message for negotiation, but there doesn’t seem to be a lot of reason for it - using the same namespace for negotiation seems fine too. In any case, this is one of those things that doesn’t matter in the slightest, and if one person volunteers to write a BIP and code, no reason they shouldn’t just decide and be allowed to run with it. Rough consensus and running code, as it were :)
    
    Matt
    
    
    > On Aug 20, 2020, at 22:37, Anthony Towns via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
    > 
    > On Fri, Aug 14, 2020 at 03:28:41PM -0400, Suhas Daftuar via bitcoin-dev wrote:
    >> In thinking about the mechanism used there, I thought it would be helpful to
    >> codify in a BIP the idea that Bitcoin network clients should ignore unknown
    >> messages received before a VERACK.  A draft of my proposal is available here
    >> [2].
    > 
    > Rather than allowing arbitrary messages, maybe it would make sense to
    > have a specific feature negotiation message, eg:
    > 
    >  VERSION ...
    >  FEATURE wtxidrelay
    >  FEATURE packagerelay
    >  VERACK
    > 
    > with the behaviour being that it's valid only between VERSION and VERACK,
    > and it takes a length-prefixed-string giving the feature name, optional
    > additional data, and if the feature name isn't recognised the message
    > is ignored.
    > 
    > If we were to support a "polite disconnect" feature like Jeremy suggested,
    > it might be easier to do that for a generic FEATURE message, than
    > reimplement it for the message proposed by each new feature.
    > 
    > Cheers,
    > aj
    > 
    > _______________________________________________
    > bitcoin-dev mailing list
    > bitcoin-dev@lists•linuxfoundation.org
    > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
    
    
    ^ permalink raw reply	[flat|nested] 30+ messages in thread

  • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
        2020-08-21 14:15   ` lf-lists
      @ 2020-08-21 16:42     ` Eric Voskuil
        2020-08-21 19:50       ` Jeremy
        0 siblings, 1 reply; 30+ messages in thread
      From: Eric Voskuil @ 2020-08-21 16:42 UTC (permalink / raw)
        To: lf-lists, Bitcoin Protocol Discussion
      
      I’m pretty sure one can run whatever they want even without a BIP. There is nobody here to do any “allowing”. On the other hand, standards development is tedious for good reason.
      
      Generally speaking, overloading is a primary source of complexity (creating more branches in code and human explanation). Nevertheless this is verack payload, no additional messages are necessary or helpful.
      
      e
      
      > On Aug 21, 2020, at 07:15, Matt Corallo via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
      > 
      > Sure, we could do a new message for negotiation, but there doesn’t seem to be a lot of reason for it - using the same namespace for negotiation seems fine too. In any case, this is one of those things that doesn’t matter in the slightest, and if one person volunteers to write a BIP and code, no reason they shouldn’t just decide and be allowed to run with it. Rough consensus and running code, as it were :)
      > 
      > Matt
      > 
      > 
      >>> On Aug 20, 2020, at 22:37, Anthony Towns via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
      >>> 
      >>> On Fri, Aug 14, 2020 at 03:28:41PM -0400, Suhas Daftuar via bitcoin-dev wrote:
      >>> In thinking about the mechanism used there, I thought it would be helpful to
      >>> codify in a BIP the idea that Bitcoin network clients should ignore unknown
      >>> messages received before a VERACK.  A draft of my proposal is available here
      >>> [2].
      >> 
      >> Rather than allowing arbitrary messages, maybe it would make sense to
      >> have a specific feature negotiation message, eg:
      >> 
      >> VERSION ...
      >> FEATURE wtxidrelay
      >> FEATURE packagerelay
      >> VERACK
      >> 
      >> with the behaviour being that it's valid only between VERSION and VERACK,
      >> and it takes a length-prefixed-string giving the feature name, optional
      >> additional data, and if the feature name isn't recognised the message
      >> is ignored.
      >> 
      >> If we were to support a "polite disconnect" feature like Jeremy suggested,
      >> it might be easier to do that for a generic FEATURE message, than
      >> reimplement it for the message proposed by each new feature.
      >> 
      >> Cheers,
      >> aj
      >> 
      >> _______________________________________________
      >> bitcoin-dev mailing list
      >> bitcoin-dev@lists•linuxfoundation.org
      >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
      > _______________________________________________
      > bitcoin-dev mailing list
      > bitcoin-dev@lists•linuxfoundation.org
      > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
      
      
      ^ permalink raw reply	[flat|nested] 30+ messages in thread

    • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
          2020-08-21 16:42     ` Eric Voskuil
        @ 2020-08-21 19:50       ` Jeremy
          2020-08-21 20:45         ` Matt Corallo
          0 siblings, 1 reply; 30+ messages in thread
        From: Jeremy @ 2020-08-21 19:50 UTC (permalink / raw)
          To: Eric Voskuil, Bitcoin Protocol Discussion
        
        [-- Attachment #1: Type: text/plain, Size: 2801 bytes --]
        
        I have a proposal:
        
        Protocol >= 70016 cease to send or process VERACK, and instead use
        HANDSHAKEACK, which is completed after feature negotiation.
        
        This should make everyone happy/unhappy, as in a new protocol number it's
        fair game to change these semantics to be clear that we're acking more than
        version.
        
        I don't care about when or where these messages are sequenced overall, it
        seems to have minimal impact. If I had free choice, I slightly agree with
        Eric that verack should come before feature negotiation, as we want to
        divorce the idea that protocol number and feature support are tied.
        
        But once this is done, we can supplant Verack with HANDSHAKENACK or
        HANDSHAKEACK to signal success or failure to agree on a connection. A NACK
        reason (version too high/low or an important feature missing) could be
        optional. Implicit NACK would be disconnecting, but is discouraged because
        a peer doesn't know if it should reconnect or the failure was intentional.
        
        ------
        
        AJ: I think I generally do prefer to have a FEATURE wrapper as you
        suggested, or a rule that all messages in this period are interpreted as
        features (and may be redundant with p2p message types -- so you can
        literally just use the p2p message name w/o any data).
        
        I think we would want a semantic (which could be based just on message
        names, but first-class support would be nice) for ACKing that a feature is
        enabled. This is because a transcript of:
        
        NODE0:
        FEATURE A
        FEATURE B
        VERACK
        
        NODE1:
        FEATURE A
        VERACK
        
        It remains unclear if Node 1 ignored B because it's an unknown feature, or
        because it is disabled. A transcript like:
        
        NODE0:
        FEATURE A
        FEATURE B
        FEATURE C
        ACK A
        VERACK
        
        NODE1:
        FEATURE A
        ACK A
        NACK B
        VERACK
        
        would make it clear that A and B are known, B is disabled, and C is
        unknown. C has 0 support, B Node 0 should support inbound messages but
        knows not to send to Node 1, and A has full bilateral support. Maybe
        instead it could a message FEATURE SEND A and FEATURE RECV A, so we can
        make the split explicit rather than inferred from ACK/NACK.
        
        
        ------
        
        I'd also propose that we add a message which is SYNC, which indicates the
        end of a list of FEATURES and a request to send ACKS or NACKS back (which
        are followed by a SYNC). This allows multi-round negotiation where based on
        the presence of other features, I may expand the set of features I am
        offering. I think you could do without SYNC, but there are more edge cases
        and the explicitness is nice given that this already introduces future
        complexity.
        
        This multi-round makes it an actual negotiation rather than a pure
        announcement system. I don't think it would be used much in the near term,
        but it makes sense to define it correctly now. Build for the future and
        all...
        
        
        
        --
        @JeremyRubin <https://twitter.com/JeremyRubin>
        <https://twitter.com/JeremyRubin>
        
        [-- Attachment #2: Type: text/html, Size: 8900 bytes --]
        
        ^ permalink raw reply	[flat|nested] 30+ messages in thread

      • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
            2020-08-21 19:50       ` Jeremy
          @ 2020-08-21 20:45         ` Matt Corallo
            2020-08-21 21:08           ` Jeremy
            2020-08-23 17:45           ` Eric Voskuil
            0 siblings, 2 replies; 30+ messages in thread
          From: Matt Corallo @ 2020-08-21 20:45 UTC (permalink / raw)
            To: Jeremy, Eric Voskuil, Bitcoin Protocol Discussion
          
          This seems to be pretty overengineered. Do you have a specific use-case in mind for anything more than simply continuing 
          the pattern we've been using of sending a message indicating support for a given feature? If we find some in the future, 
          we could deploy something like this, though the current proposal makes it possible to do it on a per-feature case.
          
          The great thing about Suhas' proposal is the diff is about -1/+1 (not including tests), while still getting all the 
          flexibility we need. Even better, the code already exists.
          
          Matt
          
          On 8/21/20 3:50 PM, Jeremy wrote:
          > I have a proposal:
          > 
          > Protocol >= 70016 cease to send or process VERACK, and instead use HANDSHAKEACK, which is completed after feature 
          > negotiation.
          > 
          > This should make everyone happy/unhappy, as in a new protocol number it's fair game to change these semantics to be 
          > clear that we're acking more than version.
          > 
          > I don't care about when or where these messages are sequenced overall, it seems to have minimal impact. If I had free 
          > choice, I slightly agree with Eric that verack should come before feature negotiation, as we want to divorce the idea 
          > that protocol number and feature support are tied.
          > 
          > But once this is done, we can supplant Verack with HANDSHAKENACK or HANDSHAKEACK to signal success or failure to agree 
          > on a connection. A NACK reason (version too high/low or an important feature missing) could be optional. Implicit NACK 
          > would be disconnecting, but is discouraged because a peer doesn't know if it should reconnect or the failure was 
          > intentional.
          > 
          > ------
          > 
          > AJ: I think I generally do prefer to have a FEATURE wrapper as you suggested, or a rule that all messages in this period 
          > are interpreted as features (and may be redundant with p2p message types -- so you can literally just use the p2p 
          > message name w/o any data).
          > 
          > I think we would want a semantic (which could be based just on message names, but first-class support would be nice) for 
          > ACKing that a feature is enabled. This is because a transcript of:
          > 
          > NODE0:
          > FEATURE A
          > FEATURE B
          > VERACK
          > 
          > NODE1:
          > FEATURE A
          > VERACK
          > 
          > It remains unclear if Node 1 ignored B because it's an unknown feature, or because it is disabled. A transcript like:
          > 
          > NODE0:
          > FEATURE A
          > FEATURE B
          > FEATURE C
          > ACK A
          > VERACK
          > 
          > NODE1:
          > FEATURE A
          > ACK A
          > NACK B
          > VERACK
          > 
          > would make it clear that A and B are known, B is disabled, and C is unknown. C has 0 support, B Node 0 should support 
          > inbound messages but knows not to send to Node 1, and A has full bilateral support. Maybe instead it could a message 
          > FEATURE SEND A and FEATURE RECV A, so we can make the split explicit rather than inferred from ACK/NACK.
          > 
          > 
          > ------
          > 
          > I'd also propose that we add a message which is SYNC, which indicates the end of a list of FEATURES and a request to 
          > send ACKS or NACKS back (which are followed by a SYNC). This allows multi-round negotiation where based on the presence 
          > of other features, I may expand the set of features I am offering. I think you could do without SYNC, but there are more 
          > edge cases and the explicitness is nice given that this already introduces future complexity.
          > 
          > This multi-round makes it an actual negotiation rather than a pure announcement system. I don't think it would be used 
          > much in the near term, but it makes sense to define it correctly now. Build for the future and all...
          > 
          > 
          > 
          > --
          > @JeremyRubin <https://twitter.com/JeremyRubin><https://twitter.com/JeremyRubin>
          
          
          ^ permalink raw reply	[flat|nested] 30+ messages in thread

        • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
              2020-08-21 20:45         ` Matt Corallo
            @ 2020-08-21 21:08           ` Jeremy
              2020-08-21 21:17             ` Jeremy
              2020-08-21 21:17             ` Eric Voskuil
              2020-08-23 17:45           ` Eric Voskuil
              1 sibling, 2 replies; 30+ messages in thread
            From: Jeremy @ 2020-08-21 21:08 UTC (permalink / raw)
              To: Matt Corallo; +Cc: Bitcoin Protocol Discussion
            
            [-- Attachment #1: Type: text/plain, Size: 4018 bytes --]
            
            Actually we already have service bits (which are sadly limited) which allow
            negotiation of non bilateral feature support, so this would supercede that.
            --
            @JeremyRubin <https://twitter.com/JeremyRubin>
            <https://twitter.com/JeremyRubin>
            
            
            On Fri, Aug 21, 2020 at 1:45 PM Matt Corallo <lf-lists@mattcorallo•com>
            wrote:
            
            > This seems to be pretty overengineered. Do you have a specific use-case in
            > mind for anything more than simply continuing
            > the pattern we've been using of sending a message indicating support for a
            > given feature? If we find some in the future,
            > we could deploy something like this, though the current proposal makes it
            > possible to do it on a per-feature case.
            >
            > The great thing about Suhas' proposal is the diff is about -1/+1 (not
            > including tests), while still getting all the
            > flexibility we need. Even better, the code already exists.
            >
            > Matt
            >
            > On 8/21/20 3:50 PM, Jeremy wrote:
            > > I have a proposal:
            > >
            > > Protocol >= 70016 cease to send or process VERACK, and instead use
            > HANDSHAKEACK, which is completed after feature
            > > negotiation.
            > >
            > > This should make everyone happy/unhappy, as in a new protocol number
            > it's fair game to change these semantics to be
            > > clear that we're acking more than version.
            > >
            > > I don't care about when or where these messages are sequenced overall,
            > it seems to have minimal impact. If I had free
            > > choice, I slightly agree with Eric that verack should come before
            > feature negotiation, as we want to divorce the idea
            > > that protocol number and feature support are tied.
            > >
            > > But once this is done, we can supplant Verack with HANDSHAKENACK or
            > HANDSHAKEACK to signal success or failure to agree
            > > on a connection. A NACK reason (version too high/low or an important
            > feature missing) could be optional. Implicit NACK
            > > would be disconnecting, but is discouraged because a peer doesn't know
            > if it should reconnect or the failure was
            > > intentional.
            > >
            > > ------
            > >
            > > AJ: I think I generally do prefer to have a FEATURE wrapper as you
            > suggested, or a rule that all messages in this period
            > > are interpreted as features (and may be redundant with p2p message types
            > -- so you can literally just use the p2p
            > > message name w/o any data).
            > >
            > > I think we would want a semantic (which could be based just on message
            > names, but first-class support would be nice) for
            > > ACKing that a feature is enabled. This is because a transcript of:
            > >
            > > NODE0:
            > > FEATURE A
            > > FEATURE B
            > > VERACK
            > >
            > > NODE1:
            > > FEATURE A
            > > VERACK
            > >
            > > It remains unclear if Node 1 ignored B because it's an unknown feature,
            > or because it is disabled. A transcript like:
            > >
            > > NODE0:
            > > FEATURE A
            > > FEATURE B
            > > FEATURE C
            > > ACK A
            > > VERACK
            > >
            > > NODE1:
            > > FEATURE A
            > > ACK A
            > > NACK B
            > > VERACK
            > >
            > > would make it clear that A and B are known, B is disabled, and C is
            > unknown. C has 0 support, B Node 0 should support
            > > inbound messages but knows not to send to Node 1, and A has full
            > bilateral support. Maybe instead it could a message
            > > FEATURE SEND A and FEATURE RECV A, so we can make the split explicit
            > rather than inferred from ACK/NACK.
            > >
            > >
            > > ------
            > >
            > > I'd also propose that we add a message which is SYNC, which indicates
            > the end of a list of FEATURES and a request to
            > > send ACKS or NACKS back (which are followed by a SYNC). This allows
            > multi-round negotiation where based on the presence
            > > of other features, I may expand the set of features I am offering. I
            > think you could do without SYNC, but there are more
            > > edge cases and the explicitness is nice given that this already
            > introduces future complexity.
            > >
            > > This multi-round makes it an actual negotiation rather than a pure
            > announcement system. I don't think it would be used
            > > much in the near term, but it makes sense to define it correctly now.
            > Build for the future and all...
            > >
            > >
            > >
            > > --
            > > @JeremyRubin <https://twitter.com/JeremyRubin><
            > https://twitter.com/JeremyRubin>
            >
            
            [-- Attachment #2: Type: text/html, Size: 5241 bytes --]
            
            ^ permalink raw reply	[flat|nested] 30+ messages in thread

          • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                2020-08-21 21:08           ` Jeremy
              @ 2020-08-21 21:17             ` Jeremy
                2020-08-21 22:16               ` Matt Corallo
                2020-08-21 21:17             ` Eric Voskuil
                1 sibling, 1 reply; 30+ messages in thread
              From: Jeremy @ 2020-08-21 21:17 UTC (permalink / raw)
                To: Jeremy; +Cc: Bitcoin Protocol Discussion
              
              [-- Attachment #1: Type: text/plain, Size: 1089 bytes --]
              
              As for an example of where you'd want multi-round, you could imagine a
              scenario where you have a feature A which gets bugfixed by the introduction
              of feature B, and you don't want to expose that you support A unless you
              first negotiate B. Or if you can negotiate B you should never expose A, but
              for old nodes you'll still do it if B is unknown to them. An example of
              this would be (were it not already out without a feature negotiation
              existing) WTXID/TXID relay.
              
              The SYNC primitve simply codifies what order messages should be in and when
              you're done for a phase of negotiation offering something. It can be done
              without, but then you have to be more careful to broadcast in the correct
              order and it's not clear when/if you should wait for more time before
              responding.
              
              
              On Fri, Aug 21, 2020 at 2:08 PM Jeremy <jlrubin@mit•edu> wrote:
              
              > Actually we already have service bits (which are sadly limited) which
              > allow negotiation of non bilateral feature support, so this would supercede
              > that.
              > --
              > @JeremyRubin <https://twitter.com/JeremyRubin>
              > <https://twitter.com/JeremyRubin>
              >
              
              [-- Attachment #2: Type: text/html, Size: 2020 bytes --]
              
              ^ permalink raw reply	[flat|nested] 30+ messages in thread

            • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                  2020-08-21 21:17             ` Jeremy
                @ 2020-08-21 22:16               ` Matt Corallo
                  2020-08-23 17:49                 ` Eric Voskuil
                  0 siblings, 1 reply; 30+ messages in thread
                From: Matt Corallo @ 2020-08-21 22:16 UTC (permalink / raw)
                  To: Jeremy; +Cc: Bitcoin Protocol Discussion
                
                Hmm, could that not be accomplished by simply building this into new messages? eg, send "betterprotocol", if you see a 
                verack and no "betterprotocol" from your peer, send "worseprotocol" before you send a "verack".
                
                Matt
                
                On 8/21/20 5:17 PM, Jeremy wrote:
                > As for an example of where you'd want multi-round, you could imagine a scenario where you have a feature A which gets 
                > bugfixed by the introduction of feature B, and you don't want to expose that you support A unless you first negotiate B. 
                > Or if you can negotiate B you should never expose A, but for old nodes you'll still do it if B is unknown to them. An 
                > example of this would be (were it not already out without a feature negotiation existing) WTXID/TXID relay.
                > 
                > The SYNC primitve simply codifies what order messages should be in and when you're done for a phase of negotiation 
                > offering something. It can be done without, but then you have to be more careful to broadcast in the correct order and 
                > it's not clear when/if you should wait for more time before responding.
                > 
                > 
                > On Fri, Aug 21, 2020 at 2:08 PM Jeremy <jlrubin@mit•edu <mailto:jlrubin@mit•edu>> wrote:
                > 
                >     Actually we already have service bits (which are sadly limited) which allow negotiation of non bilateral feature
                >     support, so this would supercede that.
                >     --
                >     @JeremyRubin <https://twitter.com/JeremyRubin><https://twitter.com/JeremyRubin>
                > 
                
                
                ^ permalink raw reply	[flat|nested] 30+ messages in thread

              • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                    2020-08-21 22:16               ` Matt Corallo
                  @ 2020-08-23 17:49                 ` Eric Voskuil
                    2020-08-24  9:44                   ` Suhas Daftuar
                    2020-08-24 19:58                   ` Jeremy
                    0 siblings, 2 replies; 30+ messages in thread
                  From: Eric Voskuil @ 2020-08-23 17:49 UTC (permalink / raw)
                    To: Matt Corallo; +Cc: Bitcoin Protocol Discussion
                  
                  
                  > On Aug 21, 2020, at 15:16, Matt Corallo <lf-lists@mattcorallo•com> wrote:
                  > 
                  > Hmm, could that not be accomplished by simply building this into new messages? eg, send "betterprotocol", if you see a verack and no "betterprotocol" from your peer, send "worseprotocol" before you send a "verack".
                  > 
                  > Matt
                  > 
                  >> On 8/21/20 5:17 PM, Jeremy wrote:
                  >> As for an example of where you'd want multi-round, you could imagine a scenario where you have a feature A which gets bugfixed by the introduction of feature B, and you don't want to expose that you support A unless you first negotiate B. Or if you can negotiate B you should never expose A, but for old nodes you'll still do it if B is unknown to them.
                  
                  This seems to imply a security benefit (I can’t discern any other rationale for this complexity). It should be clear that this is no more than trivially weak obfuscation and not worth complicating the protocol to achieve.
                  
                  >> An example of this would be (were it not already out without a feature negotiation existing) WTXID/TXID relay.
                  >> The SYNC primitve simply codifies what order messages should be in and when you're done for a phase of negotiation offering something. It can be done without, but then you have to be more careful to broadcast in the correct order and it's not clear when/if you should wait for more time before responding.
                  >> On Fri, Aug 21, 2020 at 2:08 PM Jeremy <jlrubin@mit•edu <mailto:jlrubin@mit•edu>> wrote:
                  >>    Actually we already have service bits (which are sadly limited) which allow negotiation of non bilateral feature
                  >>    support, so this would supercede that.
                  >>    --
                  >>    @JeremyRubin <https://twitter.com/JeremyRubin><https://twitter.com/JeremyRubin>
                  
                  
                  ^ permalink raw reply	[flat|nested] 30+ messages in thread

                • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                      2020-08-23 17:49                 ` Eric Voskuil
                    @ 2020-08-24  9:44                   ` Suhas Daftuar
                      2020-08-24 13:59                     ` G. Andrew Stone
                      2020-08-24 19:58                   ` Jeremy
                      1 sibling, 1 reply; 30+ messages in thread
                    From: Suhas Daftuar @ 2020-08-24  9:44 UTC (permalink / raw)
                      To: Eric Voskuil, Bitcoin Protocol Discussion
                    
                    [-- Attachment #1: Type: text/plain, Size: 2612 bytes --]
                    
                    Hi all,
                    
                    Thanks for the helpful discussion.
                    
                    My primary motivation in starting this thread was to establish what the
                    expectations are for new feature deployment (particularly whether the
                    protocol version should continue to be bumped or not), and I think I have
                    that answer -- different from what I proposed when I started this thread,
                    but not in a way that I think meaningfully hinders future work.  So I'm
                    happy to leave it at that and withdraw my suggestion.
                    
                    Cheers,
                    Suhas
                    
                    
                    On Sun, Aug 23, 2020 at 1:51 PM Eric Voskuil via bitcoin-dev <
                    bitcoin-dev@lists•linuxfoundation.org> wrote:
                    
                    >
                    > > On Aug 21, 2020, at 15:16, Matt Corallo <lf-lists@mattcorallo•com>
                    > wrote:
                    > >
                    > > Hmm, could that not be accomplished by simply building this into new
                    > messages? eg, send "betterprotocol", if you see a verack and no
                    > "betterprotocol" from your peer, send "worseprotocol" before you send a
                    > "verack".
                    > >
                    > > Matt
                    > >
                    > >> On 8/21/20 5:17 PM, Jeremy wrote:
                    > >> As for an example of where you'd want multi-round, you could imagine a
                    > scenario where you have a feature A which gets bugfixed by the introduction
                    > of feature B, and you don't want to expose that you support A unless you
                    > first negotiate B. Or if you can negotiate B you should never expose A, but
                    > for old nodes you'll still do it if B is unknown to them.
                    >
                    > This seems to imply a security benefit (I can’t discern any other
                    > rationale for this complexity). It should be clear that this is no more
                    > than trivially weak obfuscation and not worth complicating the protocol to
                    > achieve.
                    >
                    > >> An example of this would be (were it not already out without a feature
                    > negotiation existing) WTXID/TXID relay.
                    > >> The SYNC primitve simply codifies what order messages should be in and
                    > when you're done for a phase of negotiation offering something. It can be
                    > done without, but then you have to be more careful to broadcast in the
                    > correct order and it's not clear when/if you should wait for more time
                    > before responding.
                    > >> On Fri, Aug 21, 2020 at 2:08 PM Jeremy <jlrubin@mit•edu <mailto:
                    > jlrubin@mit•edu>> wrote:
                    > >>    Actually we already have service bits (which are sadly limited)
                    > which allow negotiation of non bilateral feature
                    > >>    support, so this would supercede that.
                    > >>    --
                    > >>    @JeremyRubin <https://twitter.com/JeremyRubin><
                    > https://twitter.com/JeremyRubin>
                    > _______________________________________________
                    > bitcoin-dev mailing list
                    > bitcoin-dev@lists•linuxfoundation.org
                    > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
                    >
                    
                    [-- Attachment #2: Type: text/html, Size: 3675 bytes --]
                    
                    ^ permalink raw reply	[flat|nested] 30+ messages in thread

                  • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                        2020-08-24  9:44                   ` Suhas Daftuar
                      @ 2020-08-24 13:59                     ` G. Andrew Stone
                        0 siblings, 0 replies; 30+ messages in thread
                      From: G. Andrew Stone @ 2020-08-24 13:59 UTC (permalink / raw)
                        To: Suhas Daftuar, Bitcoin Protocol Discussion
                      
                      [-- Attachment #1: Type: text/plain, Size: 4157 bytes --]
                      
                      Since discussion around allowing unknown messages or not allowing them
                      seems contentious, I'd like to offer up another possibility: create a
                      single new message, XVERSION, (and bump the protocol rev) which is a
                      key-value array of arbitrary data.  Any protocol extension can then choose
                      a new key (with a 32 or 64 bit keyspace you can basically hand out prefixes
                      to any implementation that wants one)  and publish custom data via this
                      message without needing to bump the protocol rev field.  Typical "custom
                      data" would be the min and max supported version of some specific extended
                      protocol, but any data is possible since the "value" field can be
                      serialized via the same network serialization format.  It therefore doubles
                      as a "configuration" message as well as protocol extension negotiation.
                      For example, we use it to communicate the maximum unconfirmed chain a node
                      will commit to the mempool, and peers don't bother to send transactions
                      that exceed this limit.
                      
                      You can find a specification here:
                      https://gitlab.com/bitcoinunlimited/BCHUnlimited/-/blob/dev/doc/xversionmessage.md
                      
                      Code has been deployed for a long time.
                      
                      Regards,
                      Andrew
                      
                      
                      On Mon, Aug 24, 2020 at 5:44 AM Suhas Daftuar via bitcoin-dev <
                      bitcoin-dev@lists•linuxfoundation.org> wrote:
                      
                      > Hi all,
                      >
                      > Thanks for the helpful discussion.
                      >
                      > My primary motivation in starting this thread was to establish what the
                      > expectations are for new feature deployment (particularly whether the
                      > protocol version should continue to be bumped or not), and I think I have
                      > that answer -- different from what I proposed when I started this thread,
                      > but not in a way that I think meaningfully hinders future work.  So I'm
                      > happy to leave it at that and withdraw my suggestion.
                      >
                      > Cheers,
                      > Suhas
                      >
                      >
                      > On Sun, Aug 23, 2020 at 1:51 PM Eric Voskuil via bitcoin-dev <
                      > bitcoin-dev@lists•linuxfoundation.org> wrote:
                      >
                      >>
                      >> > On Aug 21, 2020, at 15:16, Matt Corallo <lf-lists@mattcorallo•com>
                      >> wrote:
                      >> >
                      >> > Hmm, could that not be accomplished by simply building this into new
                      >> messages? eg, send "betterprotocol", if you see a verack and no
                      >> "betterprotocol" from your peer, send "worseprotocol" before you send a
                      >> "verack".
                      >> >
                      >> > Matt
                      >> >
                      >> >> On 8/21/20 5:17 PM, Jeremy wrote:
                      >> >> As for an example of where you'd want multi-round, you could imagine a
                      >> scenario where you have a feature A which gets bugfixed by the introduction
                      >> of feature B, and you don't want to expose that you support A unless you
                      >> first negotiate B. Or if you can negotiate B you should never expose A, but
                      >> for old nodes you'll still do it if B is unknown to them.
                      >>
                      >> This seems to imply a security benefit (I can’t discern any other
                      >> rationale for this complexity). It should be clear that this is no more
                      >> than trivially weak obfuscation and not worth complicating the protocol to
                      >> achieve.
                      >>
                      >> >> An example of this would be (were it not already out without a feature
                      >> negotiation existing) WTXID/TXID relay.
                      >> >> The SYNC primitve simply codifies what order messages should be in and
                      >> when you're done for a phase of negotiation offering something. It can be
                      >> done without, but then you have to be more careful to broadcast in the
                      >> correct order and it's not clear when/if you should wait for more time
                      >> before responding.
                      >> >> On Fri, Aug 21, 2020 at 2:08 PM Jeremy <jlrubin@mit•edu <mailto:
                      >> jlrubin@mit•edu>> wrote:
                      >> >>    Actually we already have service bits (which are sadly limited)
                      >> which allow negotiation of non bilateral feature
                      >> >>    support, so this would supercede that.
                      >> >>    --
                      >> >>    @JeremyRubin <https://twitter.com/JeremyRubin><
                      >> https://twitter.com/JeremyRubin>
                      >> _______________________________________________
                      >> bitcoin-dev mailing list
                      >> bitcoin-dev@lists•linuxfoundation.org
                      >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
                      >>
                      > _______________________________________________
                      > bitcoin-dev mailing list
                      > bitcoin-dev@lists•linuxfoundation.org
                      > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
                      >
                      
                      [-- Attachment #2: Type: text/html, Size: 5889 bytes --]
                      
                      ^ permalink raw reply	[flat|nested] 30+ messages in thread

                  • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                      2020-08-23 17:49                 ` Eric Voskuil
                      2020-08-24  9:44                   ` Suhas Daftuar
                    @ 2020-08-24 19:58                   ` Jeremy
                      2020-08-24 20:17                     ` Eric Voskuil
                      1 sibling, 1 reply; 30+ messages in thread
                    From: Jeremy @ 2020-08-24 19:58 UTC (permalink / raw)
                      To: Eric Voskuil; +Cc: Bitcoin Protocol Discussion
                    
                    [-- Attachment #1: Type: text/plain, Size: 1454 bytes --]
                    
                    >
                    >
                    >
                    >
                    >
                    >
                    > * >> On 8/21/20 5:17 PM, Jeremy wrote: >> As for an example of where you'd
                    > want multi-round, you could imagine a scenario where you have a feature A
                    > which gets bugfixed by the introduction of feature B, and you don't want to
                    > expose that you support A unless you first negotiate B. Or if you can
                    > negotiate B you should never expose A, but for old nodes you'll still do it
                    > if B is unknown to them. This seems to imply a security benefit (I can’t
                    > discern any other rationale for this complexity). It should be clear that
                    > this is no more than trivially weak obfuscation and not worth complicating
                    > the protocol to achieve.*
                    
                    
                    The benefit is not privacy oriented and I didn't intend to imply as such.
                    The benefit is that you may only wish to expose functionality to peers
                    which support some other set of features. For example, with wtxid relay, I
                    might want to expose some additional functionality after establishing my
                    peer supports it, that peers which do not have wtxid relay should not be
                    allowed to use. The benefit over just exposing all functions is then a node
                    might be programmed to support the new feature but not wtxid relay, which
                    can lead to some incompatibilities.
                    
                    You cannot implement this logic as a purely post-hoc "advertise all and
                    then figure out what is allowed" because then you require strict
                    consistency between peers of that post-hoc feature availability implication
                    map.
                    
                    [-- Attachment #2: Type: text/html, Size: 1869 bytes --]
                    
                    ^ permalink raw reply	[flat|nested] 30+ messages in thread

                  • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                        2020-08-24 19:58                   ` Jeremy
                      @ 2020-08-24 20:17                     ` Eric Voskuil
                        2020-08-24 20:21                       ` Jeremy
                        0 siblings, 1 reply; 30+ messages in thread
                      From: Eric Voskuil @ 2020-08-24 20:17 UTC (permalink / raw)
                        To: Jeremy; +Cc: Bitcoin Protocol Discussion
                      
                      [-- Attachment #1: Type: text/plain, Size: 1841 bytes --]
                      
                      I said security, not privacy. You are in fact exposing the feature to any node that wants to negotiate for it. if you don’t want to expose the buggy feature, then disable it. Otherwise you cannot prevent peers from accessing it. Presumably peers prefer the new feature if they support it, so there is no need for this complexity.
                      
                      e
                      
                      > On Aug 24, 2020, at 12:59, Jeremy <jlrubin@mit•edu> wrote:
                      > 
                      > 
                      >> 
                      >> >> On 8/21/20 5:17 PM, Jeremy wrote:
                      >> >> As for an example of where you'd want multi-round, you could imagine a scenario where you have a feature A which gets bugfixed by the introduction of feature B, and you don't want to expose that you support A unless you first negotiate B. Or if you can negotiate B you should never expose A, but for old nodes you'll still do it if B is unknown to them.
                      >> 
                      >> This seems to imply a security benefit (I can’t discern any other rationale for this complexity). It should be clear that this is no more than trivially weak obfuscation and not worth complicating the protocol to achieve.
                      > 
                      > 
                      > The benefit is not privacy oriented and I didn't intend to imply as such. The benefit is that you may only wish to expose functionality to peers which support some other set of features. For example, with wtxid relay, I might want to expose some additional functionality after establishing my peer supports it, that peers which do not have wtxid relay should not be allowed to use. The benefit over just exposing all functions is then a node might be programmed to support the new feature but not wtxid relay, which can lead to some incompatibilities.
                      > 
                      > You cannot implement this logic as a purely post-hoc "advertise all and then figure out what is allowed" because then you require strict consistency between peers of that post-hoc feature availability implication map.
                      
                      [-- Attachment #2: Type: text/html, Size: 2554 bytes --]
                      
                      ^ permalink raw reply	[flat|nested] 30+ messages in thread

                    • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                          2020-08-24 20:17                     ` Eric Voskuil
                        @ 2020-08-24 20:21                       ` Jeremy
                          2020-08-24 20:33                         ` Eric Voskuil
                          0 siblings, 1 reply; 30+ messages in thread
                        From: Jeremy @ 2020-08-24 20:21 UTC (permalink / raw)
                          To: Eric Voskuil; +Cc: Bitcoin Protocol Discussion
                        
                        [-- Attachment #1: Type: text/plain, Size: 987 bytes --]
                        
                        On Mon, Aug 24, 2020 at 1:17 PM Eric Voskuil <eric@voskuil•org> wrote:
                        
                        > I said security, not privacy. You are in fact exposing the feature to any
                        > node that wants to negotiate for it. if you don’t want to expose the buggy
                        > feature, then disable it. Otherwise you cannot prevent peers from accessing
                        > it. Presumably peers prefer the new feature if they support it, so there is
                        > no need for this complexity.
                        >
                        >
                        >
                        I interpreted* " This seems to imply a security benefit (I can’t discern
                        any other rationale for this complexity). It should be clear that this is
                        no more than trivially weak obfuscation and not worth complicating the
                        protocol to achieve.", *to be about obfuscation and therefore privacy.
                        
                        The functionality that I'm mentioning might not be buggy, it might just not
                        support peers who don't support another feature. You can always disconnect
                        a peer who sends a message that you didn't handshake on (or maybe we should
                        elbow bump given the times).
                        
                        [-- Attachment #2: Type: text/html, Size: 1844 bytes --]
                        
                        ^ permalink raw reply	[flat|nested] 30+ messages in thread

                      • • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                            2020-08-24 20:21                       ` Jeremy
                          @ 2020-08-24 20:33                         ` Eric Voskuil
                            0 siblings, 0 replies; 30+ messages in thread
                          From: Eric Voskuil @ 2020-08-24 20:33 UTC (permalink / raw)
                            To: Jeremy; +Cc: Bitcoin Protocol Discussion
                          
                          [-- Attachment #1: Type: text/plain, Size: 1296 bytes --]
                          
                          I see no requirement for anything here apart from exchanging a list of supported “features”. Conditionally hiding a feature provides no benefit. Any peer that wants it can get it (obfuscation being weak security), and otherwise it’s a non-issue.
                          
                          e
                          
                          > On Aug 24, 2020, at 13:22, Jeremy <jlrubin@mit•edu> wrote:
                          > 
                          >> On Mon, Aug 24, 2020 at 1:17 PM Eric Voskuil <eric@voskuil•org> wrote:
                          >> I said security, not privacy. You are in fact exposing the feature to any node that wants to negotiate for it. if you don’t want to expose the buggy feature, then disable it. Otherwise you cannot prevent peers from accessing it. Presumably peers prefer the new feature if they support it, so there is no need for this complexity.
                          > 
                          > I interpreted " This seems to imply a security benefit (I can’t discern any other rationale for this complexity). It should be clear that this is no more than trivially weak obfuscation and not worth complicating the protocol to achieve.", to be about obfuscation and therefore privacy.
                          > 
                          > The functionality that I'm mentioning might not be buggy, it might just not support peers who don't support another feature. You can always disconnect a peer who sends a message that you didn't handshake on (or maybe we should elbow bump given the times).
                          
                          [-- Attachment #2: Type: text/html, Size: 2438 bytes --]
                          
                          ^ permalink raw reply	[flat|nested] 30+ messages in thread

            • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
                2020-08-21 21:08           ` Jeremy
                2020-08-21 21:17             ` Jeremy
              @ 2020-08-21 21:17             ` Eric Voskuil
                1 sibling, 0 replies; 30+ messages in thread
              From: Eric Voskuil @ 2020-08-21 21:17 UTC (permalink / raw)
                To: Jeremy; +Cc: Bitcoin Protocol Discussion
              
              [-- Attachment #1: Type: text/plain, Size: 4319 bytes --]
              
              Service bits are advertised, protocol support is not.
              
              https://en.bitcoin.it/wiki/Protocol_documentation#Network_address
              
              e
              
              > On Aug 21, 2020, at 14:08, Jeremy <jlrubin@mit•edu> wrote:
              > 
              > 
              > Actually we already have service bits (which are sadly limited) which allow negotiation of non bilateral feature support, so this would supercede that.
              > --
              > @JeremyRubin
              > 
              > 
              >> On Fri, Aug 21, 2020 at 1:45 PM Matt Corallo <lf-lists@mattcorallo•com> wrote:
              >> This seems to be pretty overengineered. Do you have a specific use-case in mind for anything more than simply continuing 
              >> the pattern we've been using of sending a message indicating support for a given feature? If we find some in the future, 
              >> we could deploy something like this, though the current proposal makes it possible to do it on a per-feature case.
              >> 
              >> The great thing about Suhas' proposal is the diff is about -1/+1 (not including tests), while still getting all the 
              >> flexibility we need. Even better, the code already exists.
              >> 
              >> Matt
              >> 
              >> On 8/21/20 3:50 PM, Jeremy wrote:
              >> > I have a proposal:
              >> > 
              >> > Protocol >= 70016 cease to send or process VERACK, and instead use HANDSHAKEACK, which is completed after feature 
              >> > negotiation.
              >> > 
              >> > This should make everyone happy/unhappy, as in a new protocol number it's fair game to change these semantics to be 
              >> > clear that we're acking more than version.
              >> > 
              >> > I don't care about when or where these messages are sequenced overall, it seems to have minimal impact. If I had free 
              >> > choice, I slightly agree with Eric that verack should come before feature negotiation, as we want to divorce the idea 
              >> > that protocol number and feature support are tied.
              >> > 
              >> > But once this is done, we can supplant Verack with HANDSHAKENACK or HANDSHAKEACK to signal success or failure to agree 
              >> > on a connection. A NACK reason (version too high/low or an important feature missing) could be optional. Implicit NACK 
              >> > would be disconnecting, but is discouraged because a peer doesn't know if it should reconnect or the failure was 
              >> > intentional.
              >> > 
              >> > ------
              >> > 
              >> > AJ: I think I generally do prefer to have a FEATURE wrapper as you suggested, or a rule that all messages in this period 
              >> > are interpreted as features (and may be redundant with p2p message types -- so you can literally just use the p2p 
              >> > message name w/o any data).
              >> > 
              >> > I think we would want a semantic (which could be based just on message names, but first-class support would be nice) for 
              >> > ACKing that a feature is enabled. This is because a transcript of:
              >> > 
              >> > NODE0:
              >> > FEATURE A
              >> > FEATURE B
              >> > VERACK
              >> > 
              >> > NODE1:
              >> > FEATURE A
              >> > VERACK
              >> > 
              >> > It remains unclear if Node 1 ignored B because it's an unknown feature, or because it is disabled. A transcript like:
              >> > 
              >> > NODE0:
              >> > FEATURE A
              >> > FEATURE B
              >> > FEATURE C
              >> > ACK A
              >> > VERACK
              >> > 
              >> > NODE1:
              >> > FEATURE A
              >> > ACK A
              >> > NACK B
              >> > VERACK
              >> > 
              >> > would make it clear that A and B are known, B is disabled, and C is unknown. C has 0 support, B Node 0 should support 
              >> > inbound messages but knows not to send to Node 1, and A has full bilateral support. Maybe instead it could a message 
              >> > FEATURE SEND A and FEATURE RECV A, so we can make the split explicit rather than inferred from ACK/NACK.
              >> > 
              >> > 
              >> > ------
              >> > 
              >> > I'd also propose that we add a message which is SYNC, which indicates the end of a list of FEATURES and a request to 
              >> > send ACKS or NACKS back (which are followed by a SYNC). This allows multi-round negotiation where based on the presence 
              >> > of other features, I may expand the set of features I am offering. I think you could do without SYNC, but there are more 
              >> > edge cases and the explicitness is nice given that this already introduces future complexity.
              >> > 
              >> > This multi-round makes it an actual negotiation rather than a pure announcement system. I don't think it would be used 
              >> > much in the near term, but it makes sense to define it correctly now. Build for the future and all...
              >> > 
              >> > 
              >> > 
              >> > --
              >> > @JeremyRubin <https://twitter.com/JeremyRubin><https://twitter.com/JeremyRubin>
              
              [-- Attachment #2: Type: text/html, Size: 5829 bytes --]
              
              ^ permalink raw reply	[flat|nested] 30+ messages in thread

          • * Re: [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup
              2020-08-21 20:45         ` Matt Corallo
              2020-08-21 21:08           ` Jeremy
            @ 2020-08-23 17:45           ` Eric Voskuil
              1 sibling, 0 replies; 30+ messages in thread
            From: Eric Voskuil @ 2020-08-23 17:45 UTC (permalink / raw)
              To: Matt Corallo; +Cc: Bitcoin Protocol Discussion
            
            
            
            > On Aug 21, 2020, at 13:45, Matt Corallo <lf-lists@mattcorallo•com> wrote:
            > 
            > This seems to be pretty overengineered.
            
            I agree. In fact all proposals I’ve seen on this are over engineered.
            
            > Do you have a specific use-case in mind for anything more than simply continuing the pattern we've been using of sending a message indicating support for a given feature?
            
            Correct me if I’m wrong, but this pattern is what the proposal aims to eliminate. There is no reason whatsoever for a message per indication. The appropriate pattern is already established in the implementation of service bits. In fact in this discussion it has been pointed out that the problem with service bits is simply too few bits.
            
            > If we find some in the future,
            
            > we could deploy something like this, though the current proposal makes it possible to do it on a per-feature case.
            
            As does any other proposal, including passage of the full set of optional sub-protocols in the verack.
            
            > The great thing about Suhas' proposal is the diff is about -1/+1 (not including tests), while still getting all the flexibility we need.
            
            > Even better, the code already exists.
            
            This is neither true nor relevant. Maybe the Segwit 2X guys should have used this argument.
            
            e
            
            
            > Matt
            > 
            >> On 8/21/20 3:50 PM, Jeremy wrote:
            >> I have a proposal:
            >> Protocol >= 70016 cease to send or process VERACK, and instead use HANDSHAKEACK, which is completed after feature negotiation.
            >> This should make everyone happy/unhappy, as in a new protocol number it's fair game to change these semantics to be clear that we're acking more than version.
            >> I don't care about when or where these messages are sequenced overall, it seems to have minimal impact. If I had free choice, I slightly agree with Eric that verack should come before feature negotiation, as we want to divorce the idea that protocol number and feature support are tied.
            >> But once this is done, we can supplant Verack with HANDSHAKENACK or HANDSHAKEACK to signal success or failure to agree on a connection. A NACK reason (version too high/low or an important feature missing) could be optional. Implicit NACK would be disconnecting, but is discouraged because a peer doesn't know if it should reconnect or the failure was intentional.
            >> ------
            >> AJ: I think I generally do prefer to have a FEATURE wrapper as you suggested, or a rule that all messages in this period are interpreted as features (and may be redundant with p2p message types -- so you can literally just use the p2p message name w/o any data).
            >> I think we would want a semantic (which could be based just on message names, but first-class support would be nice) for ACKing that a feature is enabled. This is because a transcript of:
            >> NODE0:
            >> FEATURE A
            >> FEATURE B
            >> VERACK
            >> NODE1:
            >> FEATURE A
            >> VERACK
            >> It remains unclear if Node 1 ignored B because it's an unknown feature, or because it is disabled. A transcript like:
            >> NODE0:
            >> FEATURE A
            >> FEATURE B
            >> FEATURE C
            >> ACK A
            >> VERACK
            >> NODE1:
            >> FEATURE A
            >> ACK A
            >> NACK B
            >> VERACK
            >> would make it clear that A and B are known, B is disabled, and C is unknown. C has 0 support, B Node 0 should support inbound messages but knows not to send to Node 1, and A has full bilateral support. Maybe instead it could a message FEATURE SEND A and FEATURE RECV A, so we can make the split explicit rather than inferred from ACK/NACK.
            >> ------
            >> I'd also propose that we add a message which is SYNC, which indicates the end of a list of FEATURES and a request to send ACKS or NACKS back (which are followed by a SYNC). This allows multi-round negotiation where based on the presence of other features, I may expand the set of features I am offering. I think you could do without SYNC, but there are more edge cases and the explicitness is nice given that this already introduces future complexity.
            >> This multi-round makes it an actual negotiation rather than a pure announcement system. I don't think it would be used much in the near term, but it makes sense to define it correctly now. Build for the future and all...
            >> --
            >> @JeremyRubin <https://twitter.com/JeremyRubin><https://twitter.com/JeremyRubin>
            
            
            ^ permalink raw reply	[flat|nested] 30+ messages in thread

end of thread, other threads:[~2020-08-24 20:33 UTC | newest]

Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-14 19:28 [bitcoin-dev] Generalizing feature negotiation when new p2p connections are setup Suhas Daftuar
2020-08-16 17:24 ` Jeremy
2020-08-16 19:06 ` Eric Voskuil
2020-08-17 20:40   ` Suhas Daftuar
2020-08-17 21:21     ` Eric Voskuil
2020-08-20 14:13   ` David A. Harding
2020-08-18 14:59 ` Matt Corallo
2020-08-18 16:54   ` Eric Voskuil
2020-08-18 17:26     ` Matt Corallo
2020-08-18 18:11       ` Eric Voskuil
2020-08-18 18:25         ` Matt Corallo
2020-08-18 18:56           ` Eric Voskuil
2020-08-21  2:36 ` Anthony Towns
2020-08-21  4:25   ` Eric Voskuil
2020-08-21 14:15   ` lf-lists
2020-08-21 16:42     ` Eric Voskuil
2020-08-21 19:50       ` Jeremy
2020-08-21 20:45         ` Matt Corallo
2020-08-21 21:08           ` Jeremy
2020-08-21 21:17             ` Jeremy
2020-08-21 22:16               ` Matt Corallo
2020-08-23 17:49                 ` Eric Voskuil
2020-08-24  9:44                   ` Suhas Daftuar
2020-08-24 13:59                     ` G. Andrew Stone
2020-08-24 19:58                   ` Jeremy
2020-08-24 20:17                     ` Eric Voskuil
2020-08-24 20:21                       ` Jeremy
2020-08-24 20:33                         ` Eric Voskuil
2020-08-21 21:17             ` Eric Voskuil
2020-08-23 17:45           ` Eric Voskuil

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox