Subject: RE: [bitcoindev] The Tragic Tale of BIP30

From: <[email-masked]>
Date: Sun, 27 Apr 2025 17:01:00 -0400

> The top checkpoint is consensus for over 11 years

Correction: the block of the top checkpoint has been confirmed for over 11 years.

e

> -----Original Message-----
> From: [email-masked] <[email-masked]>
> Sent: Sunday, April 27, 2025 2:30 PM
> To: 'Ruben Somsen' <[email-masked]>; [email-masked]
> Subject: RE: [bitcoindev] The Tragic Tale of BIP30
> 
> > The problem occurs when we reorg back to a point between block 91880
> > and 91722. When we rewind the blockchain, previously created outputs
> > get removed from the UTXO set again.
> 
> Consider that a UTXO set (accumulator) is an implementation detail. One of its
> underlying assumptions is that a given txid cannot repeat in confirmed blocks.
> This assumption did not hold, and the bip30 workaround for the two
> exceptions failed to consider the effect of the above reorg given a UTXO set
> design. Instead of removing the second instance, it removes all instances.
> 
> > we could fix the bug by no longer removing the coinbase transaction in case
> of a reorg of block 91880 and 91842.
> 
> IMO this would be the most reasonable resolution, as it would produce the
> outcome that in fact exists independent of the UTXO set. The previous blocks
> still actually have the first instance of each duplicated coinbase.
> 
> This is also inconsequential from a performance standpoint. It only affects
> reorganization (infrequent), and only proceeds for these two specific blocks
> (rare). Furthermore these two specific blocks are already exceptions and the
> implementation is trivial.
> 
> > However, it seems this never occurred.
> 
> Correct, only the exception coinbases have been duplicated in the current
> strong chain.
> 
> > Aside from checking for coinbase uniqueness, we could also check that
> > the coinbase will not conflict with any future coinbases (i.e. not
> > conflict with
> > BIP34 as well as the Consensus Cleanup BIP).
> 
> The relationship between BIP34 and BIP30 is also a bit sordid, but the
> presumption is that the Consensus Cleanup would resolve the existing flaw in
> BIP34, and that the combination would effectively obsolete BIP30. Under the
> assumption that this does in fact produce the intended outcome - that BIP34
> (presently) Consensus Cleanup (forever) makes further coinbase duplication
> impossible, BIP30 can remain deactivated to the extent that these are active.
> Nothing additional is required to avoid the presumably inefficient BIP30
> checks.
> 
> > Once we fully [remove the checkpoints][3], the bug becomes theoretically
> (not practically) exploitable.
> 
> I would never advocate for adding more, but I'm not aware of any compelling
> argument to hard fork out the existing checkpoints. The top checkpoint is
> consensus for over 11 years and materially reduces the validation cost of
> 295,000 blocks.
> 
> > Doing this until block 227931 results in a modest ~7MB cache. However,
> > BIP30 might not deactivate, in which case we'd have an ever-growing cache.
> > This is solvable as follows....
> 
> This is a consequence of the presumed removal of checkpoints above BIP34
> activation height. IOW, removing the checkpoints makes it necessary to
> validate BIP30 until BIP34 activates (block 227,931). The obvious solution to
> this problem is to not create the problem in the first place.
> 
> e
> 
> > -----Original Message-----
> > From: [email-masked] <[email-masked]> On
> > Behalf Of Ruben Somsen
> > Sent: Sunday, April 27, 2025 12:45 PM
> > To: [email-masked]
> > Subject: [bitcoindev] The Tragic Tale of BIP30
> >
> > Markdown version:
> >
> https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b
> > 7c
> >
> > ### Introduction
> >
> > In my recent exploration of [SwiftSync][1], I came to the realization
> > that [BIP30][2] has an unresolved consensus bug. It seems this bug
> > can't be triggered without a reorg back to 2010, so its seriousness is
> > debatable. We currently have checkpoints up to 2013, preventing such a
> > reorg. Once we fully [remove the checkpoints][3], the bug becomes
> > theoretically (not practically) exploitable.
> >
> > BIP30 is also a bit of an odd duck in terms of consensus checks in
> > that it involves the entire UTXO set without being related to the spending of
> inputs.
> > This is inefficient, and complicates the implementation of alternative
> > validation methods such as utreexo, SwiftSync and quite possibly ZKP
> > systems such as ZeroSync. It would be nice if we could sunset BIP30
> altogether.
> >
> > Without necessarily advocating for action (the status quo seems quite
> > tenable), I'd like to lay out possible solutions for both and open up
> > the discussion.
> >
> > ### 1. Consensus bug
> >
> > There are two duplicate transactions that BIP30 treats like
> > exceptions. The last duplicate was the coinbase transaction in block
> > 91880. When this transaction gets processed, the coinbase transaction
> > in block 91722 is overwritten. The other instance occurs between these two
> blocks (91812, 91842).
> >
> > The problem occurs when we reorg back to a point between block 91880
> > and 91722. When we rewind the blockchain, previously created outputs
> > get removed from the UTXO set again. As a result, the overwritten
> > output disappears from the UTXO set completely. A node that never
> > witnessed the reorg, however, will still have the UTXO in its set. If
> > subsequently the UTXO is ever spent, this would result in a fork.
> >
> > #### Solution A
> >
> > We could enforce that no reorg can take place between block 91722 and
> > 91880 - you'd either have to reorg all of them, or none. This ensures
> > both reorged and fresh nodes will not have the problematic outputs in
> > their UTXO set. Considering this is only ~160 blocks at the low mining
> > difficulty of 2010, this wouldn't be a big constraint.
> >
> > #### Solution B
> >
> > When discussing my findings with Sjors Provoost, he pointed out that
> > the removal of the checkpoints (which can be seen as a hard fork)
> > [that is being considered][3] also presents a window of opportunity to
> > change the pre- checkpoint consensus rules - we could fix the bug by
> > no longer removing the coinbase transaction in case of a reorg of
> > block 91880 and 91842. Aside from that, Sjors' observation also opens
> > up the question whether there are other
> > pre-2013 consensus changes we'd want to consider making.
> >
> > ### 2. Sunsetting BIP30's UTXO set check
> >
> > BIP30 is currently active from genesis until [BIP34][4] activates at
> > block height
> > 227931 (March 2013). If this block is reorged out, BIP30 remains
> > active indefinitely. BIP34 has issues of its own that are being
> > addressed in the [Consensus Cleanup BIP][5] - you could go and read
> > that, I won't go into too much detail here.
> >
> > Technically, BIP30 only prevents duplicate _unspent_ outputs. It does
> > this by checking for each output whether it's already in the UTXO set
> > (this is the inefficient part), and rejecting the block if it is. The
> > two 2010 duplicates are hard-coded in as exceptions. Under these
> > rules, spending an output and recreating it is allowed. However, it seems this
> never occurred.
> >
> > One last point to address is why BIP34 gets deactivated if block
> > 227931 is reorged out. The reason for this is because otherwise it'd
> > open the door to possibly creating outputs prior to BIP34's activation
> > that will conflict with BIP34's rules for ensuring coinbase
> > transaction uniqueness (the exact issue the Consensus Cleanup is seeking to
> resolve).
> >
> > Ideally, it'd be nice to be able to sunset the BIP30 UTXO set check
> > completely, ensuring it's no longer required, even in case of a reorg.
> >
> > #### Solution
> >
> > Given that we have no duplicates, barring the two exceptions, we could
> > replace the inefficient BIP30 UTXO set check with a coinbase
> > uniqueness check. We simply cache the coinbase TXIDs and ensure there are
> no duplicates.
> > Doing this until block 227931 results in a modest ~7MB cache. However,
> > BIP30 might not deactivate, in which case we'd have an ever-growing cache.
> > This is solvable as follows.
> >
> > Aside from checking for coinbase uniqueness, we could also check that
> > the coinbase will not conflict with any future coinbases (i.e. not
> > conflict with
> > BIP34 as well as the Consensus Cleanup BIP). This ensures BIP34 can
> > simply activate at block height 227931, regardless of whether there's a
> reorg.
> >
> > ### In closing
> >
> > These were some of the issues with BIP30 and possible solutions.
> > Regardless of whether we choose to take action, this write-up will serve as a
> reference.
> > Thanks to Antoine Poinsot, Pieter Wuille, and Sjors Provoost for the
> > discussions prior to publishing.
> >
> > -- Ruben Somsen
> >
> >
> > [1]:
> >
> https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c7813
> > 3cd
> > [2]: https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki
> > [3]: https://github.com/bitcoin/bitcoin/pull/31649
> > [4]: https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki
> > [5]: https://github.com/bitcoin/bips/pull/1800
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Bitcoin Development Mailing List" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to [email-masked]
> > <mailto:[email-masked]> .
> > To view this discussion visit
> >
> https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0Yo
> > U3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com
> >
> <https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0Y
> >
> oU3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com?utm_medium=em
> > ail&utm_source=footer> .
> 

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email-masked].
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/000201dbb7b7%247af02be0%2470d083a0%24%40voskuil.org.