* [bitcoin-dev] death to the mempool, long live the mempool
@ 2021-10-26 2:56 lisa neigut
2021-10-26 8:02 ` ZmnSCPxj
` (6 more replies)
0 siblings, 7 replies; 14+ messages in thread
From: lisa neigut @ 2021-10-26 2:56 UTC (permalink / raw)
To: bitcoin-dev
[-- Attachment #1: Type: text/plain, Size: 2667 bytes --]
Hi all,
In a recent conversation with @glozow, I had the realization that the
mempool is obsolete and should be eliminated.
Instead, users should submit their transactions directly to mining pools,
preferably over an anonymous communication network such as tor. This can
easily be achieved by mining pools running a tor onion node for this
express purpose (or via a lightning network extension etc)
Mempools make sense in a world where mining is done by a large number of
participating nodes, eg where the block template is constructed by a
majority of the participants on the network. In this case, it is necessary
to socialize pending transaction data to all participants, as you don’t
know which participant will be constructing the winning block template.
In reality however, mempool relay is unnecessary where the majority of
hashpower and thus block template creation is concentrated in a
semi-restricted set.
Removing the mempool would greatly reduce the bandwidth requirement for
running a node, keep intentionality of transactions private until
confirmed/irrevocable, and naturally resolve all current issues inherent in
package relay and rbf rules. It also resolves the recent minimum relay
questions, as relay is no longer a concern for unmined transactions.
Provided the number of block template producing actors remains beneath, say
1000, it’d be quite feasible to publish a list of tor endpoints that nodes
can independently + directly submit their transactions to. In fact, merely
allowing users to select their own list of endpoints to use alternatively
to the mempool would be a low effort starting point for the eventual
replacement.
On the other hand, removing the mempool would greatly complicate solo
mining and would also make BetterHash proposals, which move the block
template construction away from a centralized mining pool back to the
individual miner, much more difficult. It also makes explicit the target
for DoS attacks.
A direct communication channel between block template construction venues
and transaction proposers also provides a venue for direct feedback wrt
acceptable feerates at the time, which both makes transaction confirmation
timelines less variable as well as provides block producers a mechanism for
(independently) enforcing their own minimum security budget. In other
words, expressing a minimum acceptable feerate for continued operation.
Initial feerate estimation would need to be based on published blocks, not
pending transactions (as this information would no longer be available), or
from direct interactions with block producers.
~niftynei
[-- Attachment #2: Type: text/html, Size: 3139 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 2:56 [bitcoin-dev] death to the mempool, long live the mempool lisa neigut
@ 2021-10-26 8:02 ` ZmnSCPxj
2021-10-26 8:31 ` eric
2021-10-26 14:09 ` darosior
` (5 subsequent siblings)
6 siblings, 1 reply; 14+ messages in thread
From: ZmnSCPxj @ 2021-10-26 8:02 UTC (permalink / raw)
To: lisa neigut, Bitcoin Protocol Discussion
Good morning lisa,
> Hi all,
>
> In a recent conversation with @glozow, I had the realization that the mempool is obsolete and should be eliminated.
>
> Instead, users should submit their transactions directly to mining pools, preferably over an anonymous communication network such as tor. This can easily be achieved by mining pools running a tor onion node for this express purpose (or via a lightning network extension etc)
>
> Mempools make sense in a world where mining is done by a large number of participating nodes, eg where the block template is constructed by a majority of the participants on the network. In this case, it is necessary to socialize pending transaction data to all participants, as you don’t know which participant will be constructing the winning block template.
>
> In reality however, mempool relay is unnecessary where the majority of hashpower and thus block template creation is concentrated in a semi-restricted set.
>
> Removing the mempool would greatly reduce the bandwidth requirement for running a node, keep intentionality of transactions private until confirmed/irrevocable, and naturally resolve all current issues inherent in package relay and rbf rules. It also resolves the recent minimum relay questions, as relay is no longer a concern for unmined transactions.
>
> Provided the number of block template producing actors remains beneath, say 1000, it’d be quite feasible to publish a list of tor endpoints that nodes can independently + directly submit their transactions to. In fact, merely allowing users to select their own list of endpoints to use alternatively to the mempool would be a low effort starting point for the eventual replacement.
>
> On the other hand, removing the mempool would greatly complicate solo mining and would also make BetterHash proposals, which move the block template construction away from a centralized mining pool back to the individual miner, much more difficult. It also makes explicit the target for DoS attacks.
Unfortunately, this requires that miners have a persistent identity by which they can be contacted.
While pseudonymity is possible, we all know that in practice, it can be easily pierced.
For instance, consider that the injunction against address reuse is a recognition that a persistent pseudonym is a privacy leak.
Ideally, the mining set should be as anonymous as possible, as some attacks are possible with sufficient hashpower, and making the miners keep a persistent identity by which they can be found may enable easier state co-option of mines.
The strongest man on Earth cannot destroy his enemy if he does not know who and where his enemy is; so with enemies of Bitcoin and the miners of Bitcoin.
(granted, near every darned mining pool self-identifies, sigh, wtf)
Ideally, the set of relaying nodes hides the miners.
Of course, in practice we can have a good guess of which relaying nodes are miners and which are not -- those who get blocks earlier are probably miners.
Against this, we should note that this method of identification is probabilistic and not absolute (whereas miners advertising their services so they can be contacted and given unconfirmed transactions are a *definite* flag "I am a miner").
And there is always the chance, however slim, that some node that has not been getting blocks "early" suddenly decides to buy a mining rig and start mining.
In short: what you propose is to switch to side fee markets (as I understand it).
Non-side fees are simply an anonymity layer, by which neither the miner nor the transactor need to know the identity of each other, they simply broadcast to the wider world.
This anonymity layer remains important, however, as they help maintain the fee market: https://github.com/libbitcoin/libbitcoin-system/wiki/Side-Fee-Fallacy
Ultimately, my objection here is simply that this requires miners to identify themselves.
In practice, miners already identify themselves (even though they really, really should not), so this objection may be moot at this point.
(Not to mention: something like P2Pool, as-is, would not work well in that model; you would need to implement a mempool for those as well)
Regards,
ZmnSCPxj
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 8:02 ` ZmnSCPxj
@ 2021-10-26 8:31 ` eric
2021-10-26 8:56 ` ZmnSCPxj
0 siblings, 1 reply; 14+ messages in thread
From: eric @ 2021-10-26 8:31 UTC (permalink / raw)
To: 'ZmnSCPxj', 'Bitcoin Protocol Discussion',
'lisa neigut'
Agree ZmnSCPxj
Hi lisa,
I'm all for removing it from memory. :) Did that a while ago. We just call it the transaction pool.
There will always be unconfirmed transactions floating around (even just from reorgs). Best to store them somewhere. Disk is cheap, block distribution (e.g. compact) works better if you have them already prevalidated, even if you aren't going to mine on them.
How you get them technically is not so important. There will always be a set of unconfirmed transactions, it's conceptual. But above all, anonymity is very important - on both ends. This is why transactions have integral fees. Anyone can get paid to mine, just need the txs.
Mining may be semi-restricted set is today, it may not be tomorrow. Imagine China everywhere, just like financial controls already are. That's when you see what Bitcoin can do from a security standpoint.
Treating miners as someone else is a poor security architecture. Everyone should look like a potential miner on the network, and a potential spender.
I think you are thinking of it a bit backwards. A node is a big pool of connected transactions. Block headers come along occasionally, and impose order on a subset of them.
e
> -----Original Message-----
> From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> On Behalf
> Of ZmnSCPxj via bitcoin-dev
> Sent: Tuesday, October 26, 2021 1:02 AM
> To: lisa neigut <niftynei@gmail•com>; Bitcoin Protocol Discussion <bitcoin-
> dev@lists•linuxfoundation.org>
> Subject: Re: [bitcoin-dev] death to the mempool, long live the mempool
>
>
> Good morning lisa,
>
> > Hi all,
> >
> > In a recent conversation with @glozow, I had the realization that the
> mempool is obsolete and should be eliminated.
> >
> > Instead, users should submit their transactions directly to mining pools,
> preferably over an anonymous communication network such as tor. This can
> easily be achieved by mining pools running a tor onion node for this express
> purpose (or via a lightning network extension etc)
> >
> > Mempools make sense in a world where mining is done by a large number
> of participating nodes, eg where the block template is constructed by a
> majority of the participants on the network. In this case, it is necessary to
> socialize pending transaction data to all participants, as you don’t know which
> participant will be constructing the winning block template.
> >
> > In reality however, mempool relay is unnecessary where the majority of
> hashpower and thus block template creation is concentrated in a semi-
> restricted set.
> >
> > Removing the mempool would greatly reduce the bandwidth requirement
> for running a node, keep intentionality of transactions private until
> confirmed/irrevocable, and naturally resolve all current issues inherent in
> package relay and rbf rules. It also resolves the recent minimum relay
> questions, as relay is no longer a concern for unmined transactions.
> >
> > Provided the number of block template producing actors remains beneath,
> say 1000, it’d be quite feasible to publish a list of tor endpoints that nodes can
> independently + directly submit their transactions to. In fact, merely allowing
> users to select their own list of endpoints to use alternatively to the mempool
> would be a low effort starting point for the eventual replacement.
> >
> > On the other hand, removing the mempool would greatly complicate solo
> mining and would also make BetterHash proposals, which move the block
> template construction away from a centralized mining pool back to the
> individual miner, much more difficult. It also makes explicit the target for DoS
> attacks.
>
> Unfortunately, this requires that miners have a persistent identity by which
> they can be contacted.
> While pseudonymity is possible, we all know that in practice, it can be easily
> pierced.
> For instance, consider that the injunction against address reuse is a
> recognition that a persistent pseudonym is a privacy leak.
>
> Ideally, the mining set should be as anonymous as possible, as some attacks
> are possible with sufficient hashpower, and making the miners keep a
> persistent identity by which they can be found may enable easier state co-
> option of mines.
> The strongest man on Earth cannot destroy his enemy if he does not know
> who and where his enemy is; so with enemies of Bitcoin and the miners of
> Bitcoin.
> (granted, near every darned mining pool self-identifies, sigh, wtf)
>
> Ideally, the set of relaying nodes hides the miners.
> Of course, in practice we can have a good guess of which relaying nodes are
> miners and which are not -- those who get blocks earlier are probably miners.
> Against this, we should note that this method of identification is probabilistic
> and not absolute (whereas miners advertising their services so they can be
> contacted and given unconfirmed transactions are a *definite* flag "I am a
> miner").
> And there is always the chance, however slim, that some node that has not
> been getting blocks "early" suddenly decides to buy a mining rig and start
> mining.
>
> In short: what you propose is to switch to side fee markets (as I understand it).
> Non-side fees are simply an anonymity layer, by which neither the miner nor
> the transactor need to know the identity of each other, they simply broadcast
> to the wider world.
> This anonymity layer remains important, however, as they help maintain the
> fee market: https://github.com/libbitcoin/libbitcoin-system/wiki/Side-Fee-
> Fallacy
>
>
> Ultimately, my objection here is simply that this requires miners to identify
> themselves.
> In practice, miners already identify themselves (even though they really,
> really should not), so this objection may be moot at this point.
>
> (Not to mention: something like P2Pool, as-is, would not work well in that
> model; you would need to implement a mempool for those as well)
>
> Regards,
> ZmnSCPxj
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 8:31 ` eric
@ 2021-10-26 8:56 ` ZmnSCPxj
0 siblings, 0 replies; 14+ messages in thread
From: ZmnSCPxj @ 2021-10-26 8:56 UTC (permalink / raw)
To: eric; +Cc: 'Bitcoin Protocol Discussion', 'lisa neigut'
Good morning e, and lisa,
> Agree ZmnSCPxj
>
> Hi lisa,
>
> I'm all for removing it from memory. :) Did that a while ago. We just call it the transaction pool.
>
> There will always be unconfirmed transactions floating around (even just from reorgs). Best to store them somewhere. Disk is cheap, block distribution (e.g. compact) works better if you have them already prevalidated, even if you aren't going to mine on them.
>
> How you get them technically is not so important. There will always be a set of unconfirmed transactions, it's conceptual. But above all, anonymity is very important - on both ends. This is why transactions have integral fees. Anyone can get paid to mine, just need the txs.
>
> Mining may be semi-restricted set is today, it may not be tomorrow. Imagine China everywhere, just like financial controls already are. That's when you see what Bitcoin can do from a security standpoint.
>
> Treating miners as someone else is a poor security architecture. Everyone should look like a potential miner on the network, and a potential spender.
>
> I think you are thinking of it a bit backwards. A node is a big pool of connected transactions. Block headers come along occasionally, and impose order on a subset of them.
On the subject of thinking backwards....
The current design gossips txes.
I believe much of what lisa wants would be doable by gossiping mining endpoints instead of txes.
Then transactors can connect to mining endpoints.
Tx gossip is limited by fees (which is why the RBF rules even exist in the first place).
Thus, mining endpoint gossip must be limited by something as well, such as by requiring some trivial stake of BTC.
(BTC exchanges are commonplace enough, I believe, that requiring completely new miners (i.e. those who currently own 0 BTC) to acquire some trivial stake of BTC would be feasible; for most people it would be easier to buy BTC than to acquire a mining rig and the supporting infrastructure needed for a mine.)
We could have the endpoint encoded in some sign-to-contract or pay-to-contract construction.
Miners can change their identity by spending their stake (which makes nodes drop their endpoint record).
Then, they can use now-common anonymity techniques --- mostly CoinJoin, but also the upcoming CoinSwap implementation --- to acquire a new stake whose identity is not easily traceable to the previous stake.
(This is not proof-of-stake, BTW --- the stake only attests the mining endpoint (in much the same way published Lightning channels are attested by their funding tx outpoints), and has no effect on block validity, only on gossiping of mining endpoints.)
The advantage here is that we expect the set of miner identities to change less often than the set of txes, thus reducing global bandwidth usage,
Against the above, we must notice that the anonymity-preserving regular changing of staked identity is more expensive than having a persistent identity.
WE should really design systems where anonymity-preservation should be as cheap as possible, but onchain activity is no longer cheap at all, given the growing importance of Bitcoin.
--
Also:
> A direct communication channel between block template construction venues and transaction proposers also provides a venue for direct feedback wrt acceptable feerates at the time, which both makes transaction confirmation timelines less variable
Unless you contact ***all*** miners globally, there is always some non-zero probability that one of the miners you did *not* contact (and thus does not have your tx, and thus will not be able to confirm your tx) gets the next block.
Since miners can enter and leave the network at any time, it is entirely possible that this mechanism *increases* variability rather than decreases it.
Regards,
ZmnSCPxj
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 2:56 [bitcoin-dev] death to the mempool, long live the mempool lisa neigut
2021-10-26 8:02 ` ZmnSCPxj
@ 2021-10-26 14:09 ` darosior
2021-10-26 16:38 ` ZmnSCPxj
2021-10-26 16:26 ` Pieter Wuille
` (4 subsequent siblings)
6 siblings, 1 reply; 14+ messages in thread
From: darosior @ 2021-10-26 14:09 UTC (permalink / raw)
To: lisa neigut, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 5764 bytes --]
Hi Niftynei,
I share the concerns raised about direct connections to mining pools being a centralization pressure: de-anonymization and an inevitable higher barrier to entry. Making it more difficult to reach smaller miners is another one.
Regarding fee estimation you state:
> Initial feerate estimation would need to be based on published blocks, not pending transactions (as this information would no longer be available)
The current fee estimation algorithm uses both, not only the pending transactions (game-able). Although i agree that past-block(s) based fee estimation isn't that bad, it's worth mentioning that not tracking the confirmation time of relayed transactions drops the ability to have a target in the estimation. That is it's good enough for time-sensitive transactions where you always target the next block but not for other usages which usually target a few dozen of blocks in the future.
However, as we discussed recently, i do believe their is a failure mode here. On one hand, directly connecting to pools is already possible today and pretty effective given the current mining centralization. On the other hand, it's not possible for most pre-signed txs protocols to reliably (securely) use the Bitcoin tx relay network today to propagate time-sensitive transactions. Furthermore, even if these are fixed (eg via package-relay for (today's) Lightning Network) it seems like there is a stark contrast between what "L2 [0] protocols" need and what regular node operators can reasonably offer. A node operator is incentivized to relay transactions to:
- have more privacy *if* they use their node to broadcast transactions (make it less distinguishable which relayed transaction comes from the wallet)
- provide fee estimates *if* they need them
- avoid bandwidth spikes on block connection (compact block relay)
L2s would ideally need the tx relaying nodes to do more computation and lift their DOS mitigations for all miner-incentive-compatible transactions to eventually be relayed to most of the miners. One obvious instance of such a dilemma is the RBF rules.
Such protocols getting increasingly developed and used create a strong incentive for their users/stakeholders to directly connect to mining pools [1], with all the consequences for the network mentioned in the replies to your mail and elsewhere.
Before we get to this, i think there is a strong case for an opt-in and publicly accessible "overlay" network to relay miner-incentive compatible transactions with higher DOS limits. This way the cost is not imposed to L1 node runners, and L2s can operate with more safety assumptions without (entirely) falling for centralization.
Thanks for publicly starting this discussion,
Antoine
[0] Using "L2s" for the sake of brevety, whatever it means i mean "protocols using pre-signed Bitcoin transactions which timely confirmation might be a security requirement".
[1] Wen block space insurance contracts
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
Le mardi 26 octobre 2021 à 4:56 AM, lisa neigut via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> a écrit :
> Hi all,
>
> In a recent conversation with @glozow, I had the realization that the mempool is obsolete and should be eliminated.
>
> Instead, users should submit their transactions directly to mining pools, preferably over an anonymous communication network such as tor. This can easily be achieved by mining pools running a tor onion node for this express purpose (or via a lightning network extension etc)
>
> Mempools make sense in a world where mining is done by a large number of participating nodes, eg where the block template is constructed by a majority of the participants on the network. In this case, it is necessary to socialize pending transaction data to all participants, as you don’t know which participant will be constructing the winning block template.
>
> In reality however, mempool relay is unnecessary where the majority of hashpower and thus block template creation is concentrated in a semi-restricted set.
>
> Removing the mempool would greatly reduce the bandwidth requirement for running a node, keep intentionality of transactions private until confirmed/irrevocable, and naturally resolve all current issues inherent in package relay and rbf rules. It also resolves the recent minimum relay questions, as relay is no longer a concern for unmined transactions.
>
> Provided the number of block template producing actors remains beneath, say 1000, it’d be quite feasible to publish a list of tor endpoints that nodes can independently + directly submit their transactions to. In fact, merely allowing users to select their own list of endpoints to use alternatively to the mempool would be a low effort starting point for the eventual replacement.
>
> On the other hand, removing the mempool would greatly complicate solo mining and would also make BetterHash proposals, which move the block template construction away from a centralized mining pool back to the individual miner, much more difficult. It also makes explicit the target for DoS attacks.
>
> A direct communication channel between block template construction venues and transaction proposers also provides a venue for direct feedback wrt acceptable feerates at the time, which both makes transaction confirmation timelines less variable as well as provides block producers a mechanism for (independently) enforcing their own minimum security budget. In other words, expressing a minimum acceptable feerate for continued operation.
>
> Initial feerate estimation would need to be based on published blocks, not pending transactions (as this information would no longer be available), or from direct interactions with block producers.
>
> ~niftynei
[-- Attachment #2: Type: text/html, Size: 6781 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 2:56 [bitcoin-dev] death to the mempool, long live the mempool lisa neigut
2021-10-26 8:02 ` ZmnSCPxj
2021-10-26 14:09 ` darosior
@ 2021-10-26 16:26 ` Pieter Wuille
2021-10-26 18:16 ` Gloria Zhao
` (3 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: Pieter Wuille @ 2021-10-26 16:26 UTC (permalink / raw)
To: lisa neigut, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 4586 bytes --]
On Monday, October 25th, 2021 at 10:56 PM, lisa neigut via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
> Hi all,
>
> In a recent conversation with @glozow, I had the realization that the mempool is obsolete and should be eliminated.
Hi Lisa,
I see where this idea is coming from, especially as it relates to reducing complexities around transaction relays, but I strongly believe this is throwing out the baby with the bathwater. Comments inline below.
> In reality however, mempool relay is unnecessary where the majority of hashpower and thus block template creation is concentrated in a semi-restricted set.
The *entire* reason mining and PoW exist, as opposed to having a fixed, centralized (set of) actors who decide transaction ordering, is to make the "censorship rights" of the network permissionless. It is essential that anyone can become a miner if they dislike what existing miners are doing, with income close to proportional to their investment. The existing reality isn't perfect, but it's fairly close to that. Sure, at any given point in time, a nontrivial fraction of mining power is in the hands of a few, but over time, those can, and have, changed a lot. Furthermore, if miners were to actually exercise censorship, it could quite reasonably incentivize other ecosystem players to start mining, perhaps close at cost or even at a small loss.
Your proposal, as far as I can tell, makes it *far* harder to become a miner. Ideas to provide a mechanism for miners to publish their "tx submit" URL/IP/onion on chain don't help; that's dependent on other miners to not censor the publishing. Furthermore, it gives a tremendous centralizing incentive: it's just far easier for most wallets to just submit to the largest few pools, because the cost/complexity of an additional submission is independent of the pool's hashrate, but the benefit is directly proportional to it. There would be very little incentive to submit to a sub-1% pool for anyone.
> Removing the mempool would greatly reduce the bandwidth requirement for running a node,
That's not true due to compact blocks (most transactions are relayed exactly once to every node, and not repeated in blocks), and with Erlay it will be even less the case.
> keep intentionality of transactions private until confirmed/irrevocable,
Except to miners; it's replacing socialized transparency with a few who get to see the actual details. Not the same scale obviously, but there is some similarity to banks in the existing financial system. Our privacy goals shouldn't be relying on a few trusted gatekeepers.
> and naturally resolve all current issues inherent in package relay and rbf rules. It also resolves the recent minimum relay questions, as relay is no longer a concern for unmined transactions.
There are other solutions to this, like weak blocks (miners get to relay partial PoW solutuon of say 10% of the difficulty to the network; and nodes which receive such a weak block can "forcibly" insert its transaction to their mempool, as there is evidence it's actually being worked on, while still being DoS resistant because partial PoW is still PoW).
> Provided the number of block template producing actors remains beneath, say 1000, it’d be quite feasible to publish a list of tor endpoints that nodes can independently + directly submit their transactions to. In fact, merely allowing users to select their own list of endpoints to use alternatively to the mempool would be a low effort starting point for the eventual replacement.
In this scenario, there is no incentive for miners to relay to each other. The fewer other miners know about a high fee-paying transaction, the better you as a miner.
More conceptually: it is a responsibility of the full node network to relay blocks between miners quickly, to limit how much advantage well-connected miners over less-well-connected ones have. If the network doesn't have the transactions being included in those blocks, this is *far* harder (additional roundtrips, as nodes can't reconstruct from mempools).
> A direct communication channel between block template construction venues and transaction proposers also provides a venue for direct feedback wrt acceptable feerates at the time, which both makes transaction confirmation timelines less variable as well as provides block producers a mechanism for (independently) enforcing their own minimum security budget. In other words, expressing a minimum acceptable feerate for continued operation.
Yes, it's definitely easier. That doesn't make it right.
Cheers,
--
Pieter
[-- Attachment #2: Type: text/html, Size: 5847 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 14:09 ` darosior
@ 2021-10-26 16:38 ` ZmnSCPxj
0 siblings, 0 replies; 14+ messages in thread
From: ZmnSCPxj @ 2021-10-26 16:38 UTC (permalink / raw)
To: darosior, Bitcoin Protocol Discussion; +Cc: lisa neigut
Good morning Antoine,
> However, as we discussed recently, i do believe their is a failure mode here. On one hand, directly connecting to pools is already possible today and pretty effective given the current mining centralization. On the other hand, it's not possible for most pre-signed txs protocols to reliably (securely) use the Bitcoin tx relay network today to propagate time-sensitive transactions. Furthermore, even if these are fixed (eg via package-relay for (today's) Lightning Network) it seems like there is a stark contrast between what "L2 [0] protocols" need and what regular node operators can reasonably offer. A node operator is incentivized to relay transactions to:
> - have more privacy *if* they use their node to broadcast transactions (make it less distinguishable which relayed transaction comes from the wallet)
> - provide fee estimates *if* they need them
> - avoid bandwidth spikes on block connection (compact block relay)
To be clear: it is possible to design L2 protocols such that various counterparties (whose incentives may not align perfectly) can bid to put their views of the L2 state on the blockchain.
For instance, in Lightning, you may wish to close a channel at a high feerate in order to use your onchain funds quickly, yet your channel counterparty has no similar time pressure to get their onchain funds in a usable state.
Solutions such as anchor outputs have been devised to allow each counterparty to pay fees as they see fit, however, for instance for anchor outputs the commitment transaction has to be at the minimum relay feerate.
At times of high blockchain congestion, node operators may raise the minimum feerate they are willing to relay (in order to alleviate bandwidth use), which may prevent commitment transactions from being relayed; even if a CPFP were relayed later, since the parent transaction is not propagated in the first place, the CPFP transaction spending the anchor outputs cannot propagate too.
I believe that is what you are referring to here as an example of how an L2 protocol cannot rely on the current L1 network for timely confirmation?
> L2s would ideally need the tx relaying nodes to do more computation and lift their DOS mitigations for all miner-incentive-compatible transactions to eventually be relayed to most of the miners. One obvious instance of such a dilemma is the RBF rules.
> Such protocols getting increasingly developed and used create a strong incentive for their users/stakeholders to directly connect to mining pools [1], with all the consequences for the network mentioned in the replies to your mail and elsewhere.
> Before we get to this, i think there is a strong case for an opt-in and publicly accessible "overlay" network to relay miner-incentive compatible transactions with higher DOS limits. This way the cost is not imposed to L1 node runners, and L2s can operate with more safety assumptions without (entirely) falling for centralization.
Let us imagine how such a network would operate.
It seems to me that an issue here is that *relay is free*.
And: "you get what you pay for".
Since mere relay is free (i.e. nodes do not charge any fee to merely *relay* transactions) the quality of that relay is low.
Thus, L1 node operators may insist on policies that do not support well miner-incentive transaction packages.
So the question is: who should pay for relay?
Ultimately of course the transactor pays, but it seems to me that the direct payer should be miners; transactors would offer higher fees, and miners would then pay part of those fees to relayers to get those transactions.
So, perhaps, let us consider a sketch (which is probably impossible, but may trigger other ideas):
Ideally, a relayer system (whether a single node, or some cooperating/competing overlay network of nodes) would gather a bunch of transactions in a proposed package of high-paying transactions.
Miners then offer to pay the relayer contingent on learning such a package.
The miner wants:
* To be given proof, before paying, that the package:
* Pays some certain value in fees.
* Consist of valid transactions.
* To atomically get the actual package once they pay.
The relayer wants:
* To be paid when it reveals such a package of transaction.
Transactors want:
* Fees to the relayer to be included as part of the mining fees in their transaction.
The flow would be something like this:
* A transactor has a group of transactions it wants confirmed.
* It goes to some relayer and feeds the transactions to them.
* The relayer figures out the most lucrative package (a task that may be NP-hard? Knapsack problem equivalent?).
* Miners look for relayers who have figured out the most lucrative next package.
* Miners pick the best package and pay for the package.
* Miners compete on mining.
The issues are:
* We need some way to prove that a bunch of transactions are valid, without showing the actual transactions.
* Maybe part of Utreexo or similar concepts can be used?
* We need some way to prove that a bunch of transactions pays some fee.
* Again, Utreexo? Maybe? Probably not?
* Fees = inputs - outputs, so the delta between the input UTXO set and the output UTXO set should be the fee, how do we prove that?
* We need some way to actually get the actual transaction data contingent on some PTLC or HTLC.
Hmm.
Thoughts?
Regards,
ZmnSCPxj
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 2:56 [bitcoin-dev] death to the mempool, long live the mempool lisa neigut
` (2 preceding siblings ...)
2021-10-26 16:26 ` Pieter Wuille
@ 2021-10-26 18:16 ` Gloria Zhao
2021-10-28 1:04 ` ZmnSCPxj
2021-10-26 23:44 ` Antoine Riard
` (2 subsequent siblings)
6 siblings, 1 reply; 14+ messages in thread
From: Gloria Zhao @ 2021-10-26 18:16 UTC (permalink / raw)
To: lisa neigut, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 9148 bytes --]
Hi Lisa,
Some background for people who are not familiar with mempool:
The mempool is a cache of unconfirmed transactions, designed in a way
to help miners efficiently pick the highest feerate packages to
include in new blocks. It stores more than a block's worth of
transactions because transaction volume fluctuates and any rational
miner would try to maximize the fees in their blocks; in a reorg, we
also don't want to completely forget what transactions were in the
now-stale tip.
In Bitcoin Core, full nodes keep a mempool by default. The additional
requirements for keeping a mempool are minimal (300MB storage, can be
configured to be lower) because anyone, anywhere in the world, should
be able to run a node and broadcast a Bitcoin payment without special
connectivity to some specific set of people or expensive/inaccessible
hardware. Perhaps connecting directly to miners can be a solution for
some people, but I don't think it's healthy for the network.
Some benefits of keeping a mempool as a non-mining node include:
- Fee estimation based on your node's knowledge of unconfirmed
transactions + historical data.
- Dramatically increased block validation (and thus propagation)
speed, since you cache signature and script results of transactions
before they are confirmed.
- Reduced block relay bandwidth usage (Bitcoin Core nodes use BIP152
compact block relay), as you don't need to re-request the block
transactions you already have in your mempool.
- Wallet ability to send/receive transactions that spend unconfirmed outputs.
> I had the realization that the mempool is obsolete and should be eliminated.
I assume you mean that the mempool should still exist but be turned
off for non-mining nodes. A block template producer needs to keep
unconfirmed transactions somewhere.
On Bitcoin Core today, you can use the -blocksonly config option to
ignore incoming transactions (effectively switching off your mempool),
but there are strong reasons not to do this:
- It is trivial for your peers to detect that all transactions
broadcasted by your node = from your wallet. Linking your node to your
transactions is a very bad privacy leak.
- You must query someone else for what feerate to put on your transaction.
- You can't use BIP152 compact block relay, so your network bandwidth
usage spikes at every block. You also can't cache validation results,
so your block validation speed slows down.
> Removing the mempool would greatly reduce the bandwidth requirement for running a node...
If you're having problems with your individual node's bandwidth usage,
you can also decrease the number of connections you make or turn off
incoming connections. There are efforts to reduce transaction relay
bandwidth usage network-wide [1].
> Removing the mempool would... keep intentionality of transactions private until confirmed/irrevocable...
I'm confused - what is the purpose of keeping a transaction private
until it confirms? Don't miners still see your transaction? A
confirmed transaction is not irrevocable; reorgs happen.
> Removing the mempool would... naturally resolve all current issues inherent in package relay and rbf rules.
Removing the mempool does not help with this. How does a miner decide
whether a conflicting transaction is an economically-advantageous
replacement or a DoS attack? How do you submit your CPFP if the parent
is below the mempool minimum feerate? Do they already have a different
relay/mempool implementation handling all of these problems but don't
aren't sharing it with the rest of the community?
> Initial feerate estimation would need to be based on published blocks, not pending transactions (as this information would no longer be available), or from direct interactions with block producers.
There are many reasons why using only published blocks for fee
estimates is a flawed design, including:
- The miner of a block can artificially inflate the feerate of the
transactions in their mempool simply by including a few of their own
transactions that pay extremely high feerates. This costs them
nothing, as they collect the fees.
- A miner constructs a block based on the transactions in their
mempool. Your transaction's feerate may have been enough to be
included 2 blocks ago or a week ago, but it will be compared to the
other unconfirmed transactions available to the miner now. They can
tell you what's in their mempool or what the next-block feerate is,
but you would be a fool to believe them.
See also [2],[3].
> Provided the number of block template producing actors remains beneath, say 1000, it’d be quite feasible to publish a list of tor endpoints that nodes can independently + directly submit their transactions to. In fact, merely allowing users to select their own list of endpoints to use alternatively to the mempool would be a low effort starting point for the eventual replacement.
As a thought experiment, let's imagine we have some public registry of
mining nodes' tor endpoints and we use it for this secondary
direct-to-miner transaction relay network. If the registry is
maintained (by who?) and accurate (based on whose word?), it is a
point of failure for transaction censorship and deanonymization, as
well as an additional barrier to becoming a miner, encouraging
centralization.
The other possibility is that the registry is not accurate. In fact,
unless the registry requires miners to identify themselves (which
others on this thread have already pointed out is ill-advised), this
should be treated similarly to regular addr gossip. We would never
automatically trust that the entity behind the endpoint provides the
service it advertises, is an honest node that won't simply blackhole
our transaction, or even belongs to a Bitcoin node at all.
Best,
Gloria
[1]: https://arxiv.org/pdf/1905.10518.pdf
[2]: https://bitcointechtalk.com/an-introduction-to-bitcoin-core-fee-estimation-27920880ad0
[3]: https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e9f41
On Tue, Oct 26, 2021 at 8:38 AM lisa neigut via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:
> Hi all,
>
> In a recent conversation with @glozow, I had the realization that the
> mempool is obsolete and should be eliminated.
>
> Instead, users should submit their transactions directly to mining pools,
> preferably over an anonymous communication network such as tor. This can
> easily be achieved by mining pools running a tor onion node for this
> express purpose (or via a lightning network extension etc)
>
> Mempools make sense in a world where mining is done by a large number of
> participating nodes, eg where the block template is constructed by a
> majority of the participants on the network. In this case, it is necessary
> to socialize pending transaction data to all participants, as you don’t
> know which participant will be constructing the winning block template.
>
> In reality however, mempool relay is unnecessary where the majority of
> hashpower and thus block template creation is concentrated in a
> semi-restricted set.
>
> Removing the mempool would greatly reduce the bandwidth requirement for
> running a node, keep intentionality of transactions private until
> confirmed/irrevocable, and naturally resolve all current issues inherent in
> package relay and rbf rules. It also resolves the recent minimum relay
> questions, as relay is no longer a concern for unmined transactions.
>
> Provided the number of block template producing actors remains beneath,
> say 1000, it’d be quite feasible to publish a list of tor endpoints that
> nodes can independently + directly submit their transactions to. In fact,
> merely allowing users to select their own list of endpoints to use
> alternatively to the mempool would be a low effort starting point for the
> eventual replacement.
>
> On the other hand, removing the mempool would greatly complicate solo
> mining and would also make BetterHash proposals, which move the block
> template construction away from a centralized mining pool back to the
> individual miner, much more difficult. It also makes explicit the target
> for DoS attacks.
>
> A direct communication channel between block template construction venues
> and transaction proposers also provides a venue for direct feedback wrt
> acceptable feerates at the time, which both makes transaction confirmation
> timelines less variable as well as provides block producers a mechanism for
> (independently) enforcing their own minimum security budget. In other
> words, expressing a minimum acceptable feerate for continued operation.
>
> Initial feerate estimation would need to be based on published blocks, not
> pending transactions (as this information would no longer be available), or
> from direct interactions with block producers.
>
>
> ~niftynei
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
[-- Attachment #2: Type: text/html, Size: 10289 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 2:56 [bitcoin-dev] death to the mempool, long live the mempool lisa neigut
` (3 preceding siblings ...)
2021-10-26 18:16 ` Gloria Zhao
@ 2021-10-26 23:44 ` Antoine Riard
2021-10-27 20:01 ` Peter Todd
2021-10-27 8:44 ` LORD HIS EXCELLENCY JAMES HRMH
2021-10-27 23:05 ` yanmaani
6 siblings, 1 reply; 14+ messages in thread
From: Antoine Riard @ 2021-10-26 23:44 UTC (permalink / raw)
To: lisa neigut, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 6082 bytes --]
Hi Lisa,
Network mempools constitute a blockspace marketplace where block demand
meets the offer in real-time. Block producers are acting to discover the
best feerate bids compensating for their operational costs and transaction
proposers are acting to offer the best feerate in function of their
confirmation preferences.
Of course in a distributed system like bitcoin, we can't guarantee perfect
information from the market participants. But moving away from this model
by decreasing the ability of the non-mining nodes to observe the current
demand is softening the requirements for potential attackers.
As transaction proposers are competing with each other to publish, they
have an interest to "front-run" each other by querying the pending
transactions to the block producers instead of observing only the published
blocks. Therefore good connections to
the block producers are now critical and censorship-resistance of the
mining endpoints must be guaranteed.
Such a list of endpoints couldn't be static otherwise it's an artificial
barrier to enter in the mining competition, and as such a centralization
vector. Dynamic, trust-minimized discovery of the mining endpoints assumes
an address-relay network, of which the robustness must be high enough
against sophisticated sybil attacks. One current defense mechanism in core
to achieve that is selecting outbound peers based in different /16 subnets
as it's harder for an attacker to obtain IP addresses. Replicating this
mechanism for the mining endpoints binds the mining topology to the
Internet one, which is downgrading the mining competition.
Relying on tor to guarantee the confidentiality of the transaction
announcement is raising its own issues. Flowing by default all the bitcoin
traffic over tor will change the incentive structure of tor attackers,
potentially attracting a new class of attackers able to do deanonymization
attacks, not that expensive in practice [0]. Tor bridges are another
censorship vector as the fingerprint of the bitcoin traffic (a block every
10 min, etc) make it possible to drop or delay the tor channel, in the lack
of high-bandwidth consuming "synthetic" traffic.
Further, identified mining endpoints make it easier to launch partition
attacks, where mining mempools are sent low-feerate clusters of
transactions, to prevent the replacement by a better feerate offer. This is
especially concerning for L2 nodes with time-sensitive requirements [1]
Lastly, removing the mempool won't solve the current issues inherent with
pre-signed transactions under the mempool min fee as ultimately miner's
mempools are also finite in memory and a dynamic lower bound must exist to
prevent spam. These lower bounds potentially increase after the signature
exchange of the time-sensitive transactions.
Antoine
[0] https://www.usenix.org/system/files/sec19-jansen.pdf
[1] See "The Ugly"
https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002758.html
Le mar. 26 oct. 2021 à 03:37, lisa neigut via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> a écrit :
> Hi all,
>
> In a recent conversation with @glozow, I had the realization that the
> mempool is obsolete and should be eliminated.
>
> Instead, users should submit their transactions directly to mining pools,
> preferably over an anonymous communication network such as tor. This can
> easily be achieved by mining pools running a tor onion node for this
> express purpose (or via a lightning network extension etc)
>
> Mempools make sense in a world where mining is done by a large number of
> participating nodes, eg where the block template is constructed by a
> majority of the participants on the network. In this case, it is necessary
> to socialize pending transaction data to all participants, as you don’t
> know which participant will be constructing the winning block template.
>
> In reality however, mempool relay is unnecessary where the majority of
> hashpower and thus block template creation is concentrated in a
> semi-restricted set.
>
> Removing the mempool would greatly reduce the bandwidth requirement for
> running a node, keep intentionality of transactions private until
> confirmed/irrevocable, and naturally resolve all current issues inherent in
> package relay and rbf rules. It also resolves the recent minimum relay
> questions, as relay is no longer a concern for unmined transactions.
>
> Provided the number of block template producing actors remains beneath,
> say 1000, it’d be quite feasible to publish a list of tor endpoints that
> nodes can independently + directly submit their transactions to. In fact,
> merely allowing users to select their own list of endpoints to use
> alternatively to the mempool would be a low effort starting point for the
> eventual replacement.
>
> On the other hand, removing the mempool would greatly complicate solo
> mining and would also make BetterHash proposals, which move the block
> template construction away from a centralized mining pool back to the
> individual miner, much more difficult. It also makes explicit the target
> for DoS attacks.
>
> A direct communication channel between block template construction venues
> and transaction proposers also provides a venue for direct feedback wrt
> acceptable feerates at the time, which both makes transaction confirmation
> timelines less variable as well as provides block producers a mechanism for
> (independently) enforcing their own minimum security budget. In other
> words, expressing a minimum acceptable feerate for continued operation.
>
> Initial feerate estimation would need to be based on published blocks, not
> pending transactions (as this information would no longer be available), or
> from direct interactions with block producers.
>
>
> ~niftynei
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
[-- Attachment #2: Type: text/html, Size: 7192 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 2:56 [bitcoin-dev] death to the mempool, long live the mempool lisa neigut
` (4 preceding siblings ...)
2021-10-26 23:44 ` Antoine Riard
@ 2021-10-27 8:44 ` LORD HIS EXCELLENCY JAMES HRMH
2021-10-27 23:05 ` yanmaani
6 siblings, 0 replies; 14+ messages in thread
From: LORD HIS EXCELLENCY JAMES HRMH @ 2021-10-27 8:44 UTC (permalink / raw)
To: bitcoin-dev, lisa neigut
[-- Attachment #1: Type: text/plain, Size: 3895 bytes --]
Good Afternoon,
No. This has been discussed previously and eliminated as there is no proof that the transaction can exist without population through the mempool. As a method of payment not hearing about a transaction until it is possibly mined three months later as I have experienced is non-functional, there were discussions in this mailing list. The purpose of the mempool is not gossip it is gossip and any node technically can mine if they do.
KING JAMES HRMH
Great British Empire
Regards,
The Australian
LORD HIS EXCELLENCY JAMES HRMH (& HMRH)
of Hougun Manor & Glencoe & British Empire
MR. Damian A. James Williamson
Wills
et al.
Willtech
www.willtech.com.au
www.go-overt.com
duigco.org DUIGCO API
and other projects
m. 0487135719
f. +61261470192
This email does not constitute a general advice. Please disregard this email if misdelivered.
________________________________
From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> on behalf of lisa neigut via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org>
Sent: Tuesday, 26 October 2021 1:56 PM
To: bitcoin-dev@lists•linuxfoundation.org <bitcoin-dev@lists•linuxfoundation.org>
Subject: [bitcoin-dev] death to the mempool, long live the mempool
Hi all,
In a recent conversation with @glozow, I had the realization that the mempool is obsolete and should be eliminated.
Instead, users should submit their transactions directly to mining pools, preferably over an anonymous communication network such as tor. This can easily be achieved by mining pools running a tor onion node for this express purpose (or via a lightning network extension etc)
Mempools make sense in a world where mining is done by a large number of participating nodes, eg where the block template is constructed by a majority of the participants on the network. In this case, it is necessary to socialize pending transaction data to all participants, as you don’t know which participant will be constructing the winning block template.
In reality however, mempool relay is unnecessary where the majority of hashpower and thus block template creation is concentrated in a semi-restricted set.
Removing the mempool would greatly reduce the bandwidth requirement for running a node, keep intentionality of transactions private until confirmed/irrevocable, and naturally resolve all current issues inherent in package relay and rbf rules. It also resolves the recent minimum relay questions, as relay is no longer a concern for unmined transactions.
Provided the number of block template producing actors remains beneath, say 1000, it’d be quite feasible to publish a list of tor endpoints that nodes can independently + directly submit their transactions to. In fact, merely allowing users to select their own list of endpoints to use alternatively to the mempool would be a low effort starting point for the eventual replacement.
On the other hand, removing the mempool would greatly complicate solo mining and would also make BetterHash proposals, which move the block template construction away from a centralized mining pool back to the individual miner, much more difficult. It also makes explicit the target for DoS attacks.
A direct communication channel between block template construction venues and transaction proposers also provides a venue for direct feedback wrt acceptable feerates at the time, which both makes transaction confirmation timelines less variable as well as provides block producers a mechanism for (independently) enforcing their own minimum security budget. In other words, expressing a minimum acceptable feerate for continued operation.
Initial feerate estimation would need to be based on published blocks, not pending transactions (as this information would no longer be available), or from direct interactions with block producers.
~niftynei
[-- Attachment #2: Type: text/html, Size: 5986 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 23:44 ` Antoine Riard
@ 2021-10-27 20:01 ` Peter Todd
0 siblings, 0 replies; 14+ messages in thread
From: Peter Todd @ 2021-10-27 20:01 UTC (permalink / raw)
To: Antoine Riard, Bitcoin Protocol Discussion; +Cc: lisa neigut
[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]
On Tue, Oct 26, 2021 at 07:44:45PM -0400, Antoine Riard via bitcoin-dev wrote:
> Such a list of endpoints couldn't be static otherwise it's an artificial
> barrier to enter in the mining competition, and as such a centralization
> vector. Dynamic, trust-minimized discovery of the mining endpoints assumes
> an address-relay network, of which the robustness must be high enough
> against sophisticated sybil attacks. One current defense mechanism in core
> to achieve that is selecting outbound peers based in different /16 subnets
> as it's harder for an attacker to obtain IP addresses. Replicating this
> mechanism for the mining endpoints binds the mining topology to the
> Internet one, which is downgrading the mining competition.
I think a really simple way to put it is if we didn't have the mempool, it'd be
good to create a free service that got transactions to miners in an equal
opportunity, decentralized, way. A simple flood fill scheme would be a great
way to do that... at which point you've re-invented the mempool.
Nothing wrong with people running nodes that opt-out of transaction
broadcasting, and it may even make sense for such nodes to preferentially peer
with each other. But there's always going to be a need for a scheme like the
existing mempool, so might as well just keep it.
--
https://petertodd.org 'peter'[:-1]@petertodd.org
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 2:56 [bitcoin-dev] death to the mempool, long live the mempool lisa neigut
` (5 preceding siblings ...)
2021-10-27 8:44 ` LORD HIS EXCELLENCY JAMES HRMH
@ 2021-10-27 23:05 ` yanmaani
6 siblings, 0 replies; 14+ messages in thread
From: yanmaani @ 2021-10-27 23:05 UTC (permalink / raw)
To: lisa neigut, Bitcoin Protocol Discussion
[I removed a comment regarding the moderation of this list here because
it caused for my message to be rejected]
On 2021-10-26 02:56, lisa neigut via bitcoin-dev wrote:
> [...] the mempool is obsolete and should be eliminated.
>
> Instead, users should submit their transactions directly to mining
> pools, [...]
> Mempools make sense in a world where mining is done by a large number
> of participating nodes, [...] as you don’t know which participant will
> be constructing the winning block template.
>
> In reality however, mempool relay is unnecessary where the majority of
> hashpower and thus block template creation is concentrated in a
> semi-restricted set.
It's true that there is some centralization, but this is hardly a
desirable goal that should be formally enshrined.
By that point, you might as well block people from keeping their coins
in their own wallet, on the basis that in practice mostly everyone keeps
them on the exchange.
And as the others have pointed out: even if you did hold this to be
desirable, why would removing the mempool be a good idea? The pools
would still need some way to get transactions, and a mempool seems like
an excellent way to do this.
I think most of the people here have laid out all of the other obvious
issues with the proposal.
> Removing the mempool would greatly reduce the bandwidth requirement
> for running a node
You can disable it already if you're strapped for cash. Is there a
reason why this is not adequate?
> keep intentionality of transactions private until confirmed/irrevocable
What is the "intentionality" of a transaction and why do I want to keep
it private? My transactions are 100% intentional because I am trying to
send money, and I wouldn't make them otherwise - what is a
non-intentional transaction supposed to be?
> Provided the number of block template producing actors remains
> beneath, say 1000, it’d be quite feasible to publish a list of tor
> endpoints that nodes can independently + directly submit their
> transactions to.
If nothing else, this would be a significant departure from the security
model of Bitcoin:
> The network is robust in its unstructured simplicity.
> Nodes work all at once with little coordination.
> They do not need to be identified, since messages are not routed to any
> particular place and only need to be delivered on a best effort basis.
> Nodes can leave and rejoin the network at will, accepting the
> proof-of-work chain as proof of what happened while they were gone.
If you posit that the security model should be changed, that is one
thing, but you should lay out your reasoning for this claim.
> On the other hand, removing the mempool would greatly complicate solo
> mining and would also make BetterHash proposals, which move the block
> template construction away from a centralized mining pool back to the
> individual miner, much more difficult.
I am amazed that you are intelligent enough to realize these trade-offs,
yet still made this post. Are you suggesting that you find them to be
acceptable?
> It also makes explicit the target for DoS attacks.
Perhaps the only good aspect of this proposal. Under such conditions,
denial of service attacks would be both just and desirable.
> A direct communication channel between block template construction
> venues and transaction proposers also provides a venue for direct
> feedback wrt acceptable feerates at the time, which both makes
> transaction confirmation timelines less variable as well as provides
> block producers a mechanism for (independently) enforcing their own
> minimum security budget. In other words, expressing a minimum
> acceptable feerate for continued operation.
Why couldn't they just run a website about this for anyone who cares?
Communicating two numbers can easily be done over HTTP. This technology
exists already.
> ~niftynei
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-26 18:16 ` Gloria Zhao
@ 2021-10-28 1:04 ` ZmnSCPxj
2021-11-03 10:12 ` ZmnSCPxj
0 siblings, 1 reply; 14+ messages in thread
From: ZmnSCPxj @ 2021-10-28 1:04 UTC (permalink / raw)
To: Gloria Zhao, Bitcoin Protocol Discussion; +Cc: lisa neigut
Good morning Gloria, et al,
> > Removing the mempool would... naturally resolve all current issues inherent in package relay and rbf rules.
>
> Removing the mempool does not help with this. How does a miner decide whether a conflicting transaction is an economically-advantageous replacement or a DoS attack? How do you submit your CPFP if the parent is below the mempool minimum feerate? Do they already have a different relay/mempool implementation handling all of these problems but don't aren't sharing it with the rest of the community?
This seems an important key point: *even if* miners maintain some kind of "accept any transaction!!!" endpoint, the miner still wants to have *some* policy on how to evict transactions from its pool of transactions, for the simple reason that *everyone* has limited resources, even miners.
Perhaps the issue is that eviction is done *immediately*, i.e. if a node receives a transaction below some feerate treshhold, it immediately drops the transaction.
What if instead we did the eviction lazily?
Suppose we used something like a garbage collector.
Unconfirmed transactions are objects that point to other objects (i.e. the input of a transaction "points to" another object).
"Primitive" objects are UTXOs of *confirmed* transactions, i.e. the UTXO set at the block tip.
Then, a GC algorithm would start at some roots and then terminate when it reaches primitive objects.
I describe here an algorithm based on semispace GC, but the GC algorithm space is well-studied and other algorithms may also be devised (in particular, spam is likely to match quite well with "infant mortality" concept in GC, i.e. "most objects die young", so some kind of nursery / generational GC may work better against spam in practice).
A semispace GC has two "spaces" for memory.
One is the "from-space" and the other is the "to-space".
During normal operation, the "from-space" is used and the "to-space" is empty.
(Note that we can implement a "space" as a table (`std::map`) from txid to transaction, and avoid having to *actually* copy the transaction data; the important thing is having two spaces)
There is a maximum size that from-space and to-space can be.
As we receive transactions, we allocate them on the from-space.
Once the from-space is filled, we stop operations and perform a GC cycle.
We select "roots" by ordering all transactions in the from-space, from highest-feerate to lowest-feerate (figure out some way to handle ties later, maybe put a timestamp or monotonic counter?).
Starting with the highest-feerate tx, we trace all the transactions they refer to, recursively, copying them from from-space to to-space.
We stop once the to-space is filled more than half.
At this point, we drop all transactions in the from-space that are not already in to-space, and then delete the from-space.
Then we promote the to-space as the from-space, and continue our merry way, allocating more transactions.
(Nothing prevents doing this on disk rather than in memory; xref Eric Voskuil)
Note that the algorithm operates on individual transactions, not packages of transactions.
The algorithm is vulnerable to spam where the spammer creates several large low-feerate transactions, then anchors them all using a tiny high-feerate transaction (a "tall" attack).
It is also vulnerable to spam where the spammer creates several high-feerate transactions spending the same UTXO (a "wide" attack), thus preventing anyone else from getting any transactions into the mempool.
Against these exploit, we can mitigate by *first* moving objects to a smaller "packagespace" instead of directly on the to-space.
When tracing a new root, we first move the transactions that are not already in to-space to the packagespace, then measure the aggregate fee divided by the aggregate memory usage.
If this is below, say, half the feerate of the root transaction, then we drop the packagespace (put it back into from-space) and move on to the next root.
This protects against "tall" attacks.
To protect against "wide" attacks, if the packagespace consumes a TXO that is already consumed in the to-space, we also drop the packagespace (i.e. only retain the highest-feerate version in a "wide" attack).
Once the above checks pass, we merge the packagespace into the to-space.
This algorithm means that we do not need package relay; instead, we just send transactions in the correct order (parents first, then children), and if the receiver does not need to do a GC in-between, then everything ends up in the mempool.
If the child transaction is high-fee enough to be a root transaction, and pays enough that its feerate dominates in the packagespace result, then the entire sequence will remain in the mempool.
The algorithm allows for conflicting transactions to be retained in the mempool temporarily, until the next GC triggers (at which point conflicting transactions are resolved by whoever is higher-feerate).
This is helpful since a conflicting transaction may be what ends up getting confirmed in a block from a miner whose mempool did not contain the "best" feerate transaction.
WDYT?
Regards,
ZmnSCPxj
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] death to the mempool, long live the mempool
2021-10-28 1:04 ` ZmnSCPxj
@ 2021-11-03 10:12 ` ZmnSCPxj
0 siblings, 0 replies; 14+ messages in thread
From: ZmnSCPxj @ 2021-11-03 10:12 UTC (permalink / raw)
To: ZmnSCPxj, Bitcoin Protocol Discussion; +Cc: lisa neigut
Good morning list,
> I describe here an algorithm based on semispace GC, but the GC algorithm space is well-studied and other algorithms may also be devised (in particular, spam is likely to match quite well with "infant mortality" concept in GC, i.e. "most objects die young", so some kind of nursery / generational GC may work better against spam in practice).
This "nursery GC" variation would be very similar to these "pre-mempool" / "memswamp" proposals, as well:
* https://github.com/bitcoin/bitcoin/issues/14895#issuecomment-665975441
* https://github.com/bitcoin/bitcoin/issues/14895#issuecomment-666267749
It may be better to continue thinking along those lines than to consider this GC concept.
Regards,
ZmnSCPxj
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-11-03 10:13 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-26 2:56 [bitcoin-dev] death to the mempool, long live the mempool lisa neigut
2021-10-26 8:02 ` ZmnSCPxj
2021-10-26 8:31 ` eric
2021-10-26 8:56 ` ZmnSCPxj
2021-10-26 14:09 ` darosior
2021-10-26 16:38 ` ZmnSCPxj
2021-10-26 16:26 ` Pieter Wuille
2021-10-26 18:16 ` Gloria Zhao
2021-10-28 1:04 ` ZmnSCPxj
2021-11-03 10:12 ` ZmnSCPxj
2021-10-26 23:44 ` Antoine Riard
2021-10-27 20:01 ` Peter Todd
2021-10-27 8:44 ` LORD HIS EXCELLENCY JAMES HRMH
2021-10-27 23:05 ` yanmaani
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox