Re: [bitcoin-dev] On mempool policy consistency

From: Gloria Zhao <gloriajzhao@gmail•com>
To: Anthony Towns <aj@erisian•com.au>,
	 Bitcoin Protocol Discussion
	<bitcoin-dev@lists•linuxfoundation.org>
Subject: Re: [bitcoin-dev] On mempool policy consistency
Date: Thu, 27 Oct 2022 13:36:47 +0100	[thread overview]
Message-ID: <CAFXO6=+Bjr_nuy+24VZR-zE3=Heii_NJxWPdiJTJW6ZgWbZ8rA@mail.gmail.com> (raw)
In-Reply-To: <Y1nIKjQC3DkiSGyw@erisian.com.au>

[-- Attachment #1: Type: text/plain, Size: 25110 bytes --]

Hi AJ,

Not going to comment on what Bitcoin Core's philosophy on mempol policy is
or should be. I want to note that I think this:

> It's also possible that this is something of a one time thing: full rbf
> has been controversial for ages, but widely liked by devs, and other
> attempts (eg making it available in knots) haven't actually achieved
> much of a result in practice. So maybe this is just a special case

is true.

> The second thing is that whatever your relay policy is, you still
> need a path all the way to miners through nodes that will accept your
> transaction at every step. If you're making your mempool more restrictive
> (eg -permitbaremultisig=0, -datacarrier=0), that's easy for you (though
> you're making life more difficult for people who do create those sorts
> of txs); but if you want a more permissive policy (package relay,
> version-3-rbf, full-rbf), you might need to do some work.

> The cutoff for that is probably something like "do 30% of listening
> nodes have a compatible policy"? If they do, then you'll have about a
> 95% chance of having at least one of your outbound peers accept your tx,
> just by random chance.

Yes, in most cases, whether Bitcoin Core is restricting or loosening
policy, the user in question is fine as long as they have a path from their
node to a miner that will accept it. This is the case for something like
-datacarriersize if the use case is putting stuff into OP_RETURN outputs,
or if they're LN and using CPFP carveout, v3, package relay, etc. But
replacement is not only a question of "will my transaction propagate" but
also, "will someone else's transaction propagate, invalidating mine" or, in
other words, "can I prevent someone else's transaction from propagating." A
zeroconf user relies on there *not* being a path from someone else's full
RBF node to a full RBF miner. This is why I think RBF is so controversial
in general, why -mempoolfullrbf on someone else's node is considered more
significant than another policy option, and why full RBF shouldn't be
compared with something like datacarriersize. I don't think past patterns
can be easily applied here, and I don't think this necessarily shows a
different "direction" in thinking about mempool policy in general.

Best,
Gloria

On Thu, Oct 27, 2022 at 12:52 AM Anthony Towns via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> Hi *,
>
> TLDR: Yes, this post is too long, and there's no TLDR. If it's any
> consolation, it took longer to write than it does to read?
>
> On Tue, Jun 15, 2021 at 12:55:14PM -0400, Antoine Riard via bitcoin-dev
> wrote:
> > Subject: Re: [bitcoin-dev] Proposal: Full-RBF in Bitcoin Core 24.0
> > I'm writing to propose deprecation of opt-in RBF in favor of full-RBF
>
> > If there is ecosystem agreement on switching to full-RBF, but 0.24 sounds
> > too early, let's defer it to 0.25 or 0.26. I don't think Core has a
> > consistent deprecation process w.r.t to policy rules heavily relied-on by
> > Bitcoin users, if we do so let sets a precedent satisfying as many folks
> as
> > we can.
>
> One precedent that seems to be being set here, which to me seems fairly
> novel for bitcoin core, is that we're about to start supporting and
> encouraging nodes to have meaningfully different mempool policies. From
> what I've seen, the baseline expectation has always been that while
> certainly mempools can and will differ, policies will be largely the same:
>
>   Firstly, there is no "the mempool". There is no global mempool. Rather
>   each node maintains its own mempool and accepts and rejects transaction
>   to that mempool using their own internal policies. Most nodes have
>   the same policies, but due to different start times, relay delays,
>   and other factors, not every node has the same mempool, although they
>   may be very similar.
>
>   -
> https://bitcoin.stackexchange.com/questions/98585/how-to-find-if-two-transactions-in-mempool-are-conflicting
>
> Up until now, the differences between node policies supported by different
> nodes running core have been quite small, with essentially the following
> options available:
>
>  -minrelaytxfee, -maxmempool - changes the lowest fee rate you'll accept
>
>  -mempoolexpiry - how long to keep txs in the mempool
>
>  -datacarrier - reject txs creating OP_RETURN outputs
>
>  -datacarriersize - maximum size of OP_RETURN data
>
>  -permitbaremultisig - prevent relay of bare multisig
>
>  -bytespersigop - changes how SIGOP accounting works for relay and
>  mining prioritisation
>
> as well as these, marked as "debug only" options (only shown with
> -help-debug):
>
>  -incrementalrelayfee - make it easier/harder to spam txs by only
>  slightly bumping the fee; marked as a "debug only" option
>
>  -dustrelayfee - make it easier/harder to create uneconomic utxos;
>  marked as a "debug only" option
>
>  -limit{descendant,ancestor}{count,size} - changes how large the
>  transaction chains can be; marked as a "debug only" option
>
> and in theory, but not available on mainnet:
>
>  -acceptnonstdtxn - relay/mine non standard transactions
>
> There's also the "prioritisetransaction" rpc, which can cause you to keep
> a low feerate transaction in your mempool longer than you might otherwise.
>
> I think that -minrelaytxfee, -maxmempool and -mempoolexpiry are the only
> ones of those options commonly set, and those only rarely result in any
> differences in the txs at the top of the mempool.
>
> There are also quite a few parameters that aren't even runtime
> configurable:
>
>  - MAX_STANDARD_TX_WEIGHT
>  - MIN_STANDARD_TX_NONWITNESS_SIZE (see also #26265)
>  - MAX_P2SH_SIGOPS (see also #26348)
>  - MAX_STANDARD_TX_SIGOPS_COST
>  - MAX_STANDARD_P2WSH_STACK_ITEMS
>  - MAX_STANDARD_P2WSH_STACK_ITEM_SIZE
>  - MAX_STANDARD_TAPSCRIPT_STACK_ITEM_SIZE
>  - MAX_STANDARD_P2WSH_SCRIPT_SIZE
>  - MAX_STANDARD_SCRIPTSIG_SIZE
>  - EXTRA_DESCENDANT_TX_SIZE_LIMIT
>  - MAX_REPLACEMENT_CANDIDATES
>
> And other plausible options aren't configurable even at compile time
> -- eg, core doesn't implement BIP 125's inherited signalling rule so
> there's no way to enable it; core doesn't allow opting out of BIP 125
> rule 3 ratchet on absolute fee; core doesn't allow CPFP carveout with
> more than 1 ancestor; core doesn't allow opting out of LOW_S checks
> (even via -acceptnonstdtxn); etc.
>
> We also naturally have different mempool policies between different
> releases: eg, expansions of policy, such as allowing OP_RETURN or
> expanding it from 40 to 80 bytes or new soft forks where old nodes won't
> relay transactions that use the new; and also occassional restrictions
> in policy, such as the LOW_S requirement.
>
>
> While supporting and encouraging different mempool polices might be new
> for core, it's not new for knots: knots changes some of these defaults
> (-permitbaremultisig defaults to false, -datacarriersize is reduced to
> 42), allows the use of -acceptnonstdtxn on mainnet, and introduces new
> options including -spkreuse and -mempoolreplacement (giving the latter
> full rbf behaviour by default). Knots also includes a `-corepolicy`
> option to make it easy to get a configuration matching core's defaults.
>
>
> I think gmaxwell's take from Feb 2015 (in the context of how restrictive
> policy on OP_RETURN data should be) was a reasonable description for
> core's approach up until now:
>
>   There is also a matter of driving competent design rather than lazy
>   first thing that works. E.g. In stealth addresses the early proposals
>   use highly inefficient single ECDH point per output instead of simply
>   pooling them. Network behavior is one of the few bits of friction
>   driving good technical design rather than "move fast, break things, and
>   force everyone else onto my way of doing thing rather than discussing
>   the design in public". No one wants to be an outright gatekeeper,
>   but the network is a shared resource and it's perfectly reasonable
>   node behavior to be stingy about the perpetual storage impact of the
>   transactions they're willing to process, especially when it comes to
>   neutral technical criteria like the amount of network irrelevant data
>   stuffed in transactions.
>
>   There is also a very clear pattern we've seen in the past where
>   people take anything the system lets them do as strong evidence that
>   they have a irrevocable right to use the system in that way, and that
>   their only responsibility-- and if their usage harms the system it's
>   the responsibility of the system to not permit it. [...
>   ...] For mitigating these risks it's optimal if transactions
>   seem as uniform and indistinguishable as reasonably possible.
>
>   - https://github.com/bitcoin/bitcoin/pull/5286#issuecomment-72564175
>
> Perhaps see also sdaftuar in Nov 2015,
>
>   To me the most important question is, is priority something that miners
>   want to use?
>
>   If a non-negligible amount of hashpower intends to use it in their
>   transaction selection, then I think it makes sense for nodes to use it
>   too, because it's generally helpful to have your mempool predict the
>   UTXO as much as possible, and for nodes to be able to have reasonable
>   fee and priority estimates (which won't happen unless they track the
>   priority transactions somehow -- I'm presuming that miners run with
>   much bigger mempools than regular nodes).
>
>   If the answer is no, then that's fine and I don't see a reason to push
>   in this direction. I sort of assumed there was enough hashpower mining
>   with priority, since last time I checked estimatepriority was still
>   giving meaningful results for low-ish blockheights, but I haven't done
>   any kind of real analysis.
>
>   - https://github.com/bitcoin/bitcoin/pull/6992#issuecomment-155969455
>
> or in June 2019,
>
>   What this PR is proposing is to get rid of a command-line option that is
>   (a) a footgun for users and (b) does not reflect what I believe to be
>   the understanding most users have, which is that [X txs] are expected
>   to propagate well on the network.
>
>   ..
>
>   I don't think this rises to the level that Luke is concerned about,
>   namely a prelude to forcing a common relay policy on all nodes. In
>   particular I do agree it makes sense that we offer some ways of
>   customizing policy parameters (eg the mempool size, min relay fee,
>   etc). Instead, I think the justification for this change is that we
>   should not support behaviors we think are harmful to the ecosystem
>   overall and have no legitimate use-case, and we should eliminate ways
>   that users might inadvertently shoot themselves in the foot.
>
>   - https://github.com/bitcoin/bitcoin/pull/16171#issuecomment-500393271
>
> (or see discussion in https://github.com/bitcoin/bitcoin/pull/7219)
>
> I don't mean to imply the above are saying "there's one way to do
> things and it's this way", or that the old way of doing things should
> necessarily be the way we keep doing things. Just that previously core
> has tended towards designing a single policy that works as well as it
> can for everyone and the ecosystem as a whole. (I'm also not saying that
> fullrbf can't work well for everyone or the ecosystem as a whole)
>
>
> By contrast, I think the most common response to pushback against the
> full rbf option has been along the lines of "it's just an option, we
> don't want to force people", eg:
>
>   Blaming the default false -mempoolfullrbf option for a full RBF network
>   would be holding Bitcoin Core developers responsible for the decisions
>   of individual node operators and miners. I don't think having the
>   option (again, default false) can directly cause a full RBF network,
>   and likewise, I don't think removing this option removes the "risk"
>   of a full RBF network.
>    - glozow
>      https://github.com/bitcoin/bitcoin/pull/26287#issuecomment-1274949400
>
>   NACK. This is a default false option.
>    - achow101
>      https://github.com/bitcoin/bitcoin/pull/26287#issuecomment-1274953204
>
>   Erecting artificial barriers to prevent or make it difficult for users
>   to do what they want to do, is not appropriate behaviour.
>    - luke-jr
>      https://github.com/bitcoin/bitcoin/pull/26287#issuecomment-1290721905
>
>   I'm in general against removing options.
>    - instagibbs
>      https://github.com/bitcoin/bitcoin/pull/26287#issuecomment-1292030700
>
> I think this differs from what core has done in the past, in that
> previously we've tried to ensure a new policy is good for everyone (or as
> nearly as it can be), and then enabled it as soon as it's implemented.
> Any options that have been added have either been to control resource
> usage in ways that don't significantly effect tx propagation, to
> allow people to revert to the old behaviour when the new behaviour is
> controversial (eg the -mempoolreplacement=0 option from 0.12 to 0.18),
> and to make it easier to test/debug the implementation.
>
> Giving people a new relay behaviour they can opt-in to when we aren't
> confident enough to turn on by default doesn't match the approach I've
> seen core take in the past.
>
>
> If this is going to be an ongoing shift in how core sees relay/mempool
> policy, I think that's significant and worth paying attention to.
>
> I don't think it's necessary to have that shift to roll out full rbf.
> The other approach would be either:
>
>  * set -mempoolfullrbf=true as the default for 24.0, and just have the
>    command line param there in case people want to do a
>    "UserRejectedMempoolPolicy" campaign to get everyone to opt-out
>
>  * revert it for now because we don't think mainnet is ready for fullrbf
>    yet, and introduce it as default true for 25.0 or 26.0 or 27.0 or
>    to activate at some scheduled date in that timeframe (potentially
>    backporting it to previous releases to help with adoption too,
>    whatever). same effect as the previous option, just with a bit more
>    advanced notice and time to prepare
>
> I don't think anyone's proposed the first (which I interpret as "most of
> us don't think mainnet is ready for fullrbf today"), but the comments
> above are all pushback by people arguing against (the first step of)
> the second approach, and they seem to be winning the day.
>
> It's also possible that this is something of a one time thing: full rbf
> has been controversial for ages, but widely liked by devs, and other
> attempts (eg making it available in knots) haven't actually achieved
> much of a result in practice. So maybe this is just a special case and
> not a precedent, and when people propose other default false options,
> there will be substantially more resistance to them being merged,
> despite all the talk about users having options that's going on right now.
>
>
> Assuming it is the change of direction it appears to be -- and all of
> the above is really just justification for that assumption -- then like
> I said, I think it's worth seriously considering what it means for people
> to choose their own relay/mempool policies and for you to expect to have
> different mempool policies to many or most of your potential peers.
>
>
> One thing maybe worth noting is that is that you can still only choose
> your policy from options that people write code for -- if it wasn't
> something you could get by running knots or compiling a rejected PR
> yourself, it won't magically become more possible now.  Presumably it
> would mean that once a PR is written, it might get better review (rather
> than being dismissed as not suitable for everyone), and there would be
> less maintenance burden than if it had to be manually rebased every
> release, though (or at least the maintenance burden would be shared
> across everyone working on the codebase).
>
>
> The second thing is that whatever your relay policy is, you still
> need a path all the way to miners through nodes that will accept your
> transaction at every step. If you're making your mempool more restrictive
> (eg -permitbaremultisig=0, -datacarrier=0), that's easy for you (though
> you're making life more difficult for people who do create those sorts
> of txs); but if you want a more permissive policy (package relay,
> version-3-rbf, full-rbf), you might need to do some work.
>
> The cutoff for that is probably something like "do 30% of listening
> nodes have a compatible policy"? If they do, then you'll have about a
> 95% chance of having at least one of your outbound peers accept your tx,
> just by random chance. If erlay allows increasing your outbound count to
> 12 connections instead of 8; that might reduce down to needing just 20%
> of listening nodes (~93%).
>
> But for cases where less than 30% (20%) of network supports your preferred
> policy, you probably need to do something cleverer.
>
> One approach is to set a service bit and preferentially peer with other
> nodes that advertise that service bit; knots does the first half of this
> for fullrbf, and both halves have been proposed for core in #25600.
> Preferential peering was previously done for the segwit deployment,
> though in that case it was necessary not just for tx propogation but
> also for ensuring block propogation, making it effectively a consensus
> critical issue.
>
> Another approach is having a separate relay network -- eg, lightning nodes
> already have a gossip network, and might want to help their own ecosystem
> by ensuring unilateral channel closes and justice transactions are quickly
> relayed. Using their own gossip network to relay the transaction around,
> and each lightning node adding it to their local bitcoind's mempool and
> allowing it to propogate (or not) from there as normal, would also be a
> way of allowing transactions to propogate well. It does mean that miners
> would either need to also participate in lightning gossip directly, or
> that miners would need to connect to *many* peers to be confident of
> seeing those transactions (eg, if only 2% of the network would see a
> tx, you'd need to make 228 connections to have a 99% chance of seeing
> the tx). You can't currently do something like this, because all the
> relay policies are also applied when adding txs to the mempool via RPC,
> and there's no convenient way to remove txs from the mempool.
>
> A case where something like that might occur is in preventing L2
> transactions from pinning attacks -- so you might have a high-fee,
> low-feerate transaction that's been widely propogated, sitting in the
> bottom of people's mempools, and you want to replace it with a smaller,
> higher-feerate transaction, but don't want to pay a higher absolute fee,
> and are thus blocked by BIP 125 rule 3. Perhaps 98% of the network is
> unwilling to deviate from BIP 125 rule 3 for you; because that would
> make it easy for random griefers to spam their mempool with large txs
> then delete them while only paying a small fee; but your L2 peers may be
> able to decode your replacement transaction and be sure that you aren't
> going to spam them, and thus will happily relay it.
>
> From a technical point-of-view, that's largely fine; the downside is it
> increases the centralisation pressure on mining: whether that's by having
> to connect to substantially more nodes, or having to parse through more
> spam, you can't just run your mining operation off a standard install
> of bitcoin core anymore, but need to actively opt-in to find all the
> weird unusual ways people are sending transactions around in order to
> actually collect as much in fees as your competitors are.
>
> That's probably moderately bad for privacy as well -- if lightning or
> coinjoins need special relay rules that most nodes haven't opted into,
> it's potentially easy to use that to find the bitcoin nodes on the
> network that are participating in those protocols, and from there to
> either identify the operator, or run a DoS attack to make it hard for you
> to keep doing what you want. Obviously if you're setting a service bit to
> get better routing, you've given up that privacy already. Likewise if the
> government or random vandals are opposed to bitcoin mining, and miners
> have to have special configuration on their nodes that distinguish them
> from regular users, then perhaps that makes it easier to find or shut
> down their operations.
>
> There are a few efficiencies to be gained from similar mempool policies as
> well: more reliable compact block reconstruction (if you're not missing
> any transactions, you avoid a round-trip) and presumably more efficient
> set reconstruction with erlay. You'll also waste less bandwidth sending
> transactions that the other node is only going to reject. Both those
> depend on how many transactions are going to rely on unusual mempool
> policies in the first place though.
>
> ariard wrote:
>
>   I know I've advocated in the past to turn RBF support by default in
>   the past. Though after gathering a lot of feedbacks, this approach
>   of offering the policy flexiblity to the interested users only and
>   favoring a full-rbf gradual deployment sounds better to me.
>
>   - https://github.com/bitcoin/bitcoin/pull/25353#issuecomment-1157137026
>
> I guess all the above leads me to think that gradual deployments of
> mempool policies are likely the worse approach: even when they're not
> hurting anyone, it makes them hard to use during the gradual phase,
> and getting around that comes with worrying compromises on privacy and
> centralisation; and when they are problematic for some, the indeterminate
> nature of a gradual deployment means it's hard to plan for when that
> risk is going to eventuate.
>
>
> Theoretically, one way to recover the good parts of core deciding on
> what's good for the network might be for people outside of core to
> recommend a mempool configuration; then core can just have an option
> to make that easy, similar to "-std=c++17" for a C++ compiler, and much
> the same as knots' "-corepolicy" option.
>
> Presuming anyone actually wants to take on that job, and listen to the
> concerns of zeroconf businesses, lightning and coinjoin devs, miners, etc;
> and can come up with something that keeps most of them happy, and that
> 70% or 90% of the network ends up just following those recommendations
> because it's easy, it works, and it's recommended by all the apps they
> want to use, then that could work great:
>
>  * miners don't need to do anything special, so there's no new
>    mining centralisation pressure
>  * miners and users don't reveal what they're doing with bitcoin by the way
>    they configure their nodes, so there's no privacy problems
>  * devs can be fairly confident in how they have to design their apps
>    in order to get their transactions to most hashpower
>  * devs don't have to add new p2p layers to make it happen
>  * at least there's someone to talk to when you're trying to figure out
>    how to make some new project possible when it's inhibited by current
>    relay policies and you don't have to try to convince everyone to
>    upgrade on your own
>  * core devs just provide options, and don't have to worry about being
>    seen as gatekeepers
>
> The "downside" in that scenario is that users/dev aren't making much
> actual use of all the choices core is offering by making different
> options available; but the upside is that that choice is at least readily
> available should whoever is coming up with these policy become out of
> step with what people actually want.
>
> One thing that might make an approach like that difficult is that core
> has historically been happy to remove options that don't seem useful
> anymore: eg the ability to turn of BIP 125 support (#16171), and priority
> transactions (#9602). Perhaps that's fine if you're trying to actively
> craft a single mempool/relay policy that's good enough for almost everyone
> (after all, it makes the code simpler and more efficient, and reduces
> the number of footguns); all you're doing is leaving a minority of people
> who want weird things to run a fork, and that's going to happen anyway.
>
> But if people are following policy developed outside of core, core
> might well disagree with them and decide "no that's a stupid policy,
> no one should do that" and remove some feature that others thing should
> continue to be normal. Beyond the examples above, there's already talk of
> removing the ability to disable fullrbf support in #26305, for instance.
> If that happens, then the people maintaining the policy will instead
> end up maintaining an entire fork of bitcoin core, and all we've done
> is transition to people running software from a different repo, and a
> different set of maintainers.
>
> If we're really going to a world where core's eager to add new options,
> and reluctant to remove them, at least if anyone at all finds them
> interesting, that's presumably a non-issue, though.
>
> Cheers,
> aj
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 29414 bytes --]