* [bitcoindev] Removing checkpoints in Bitcoin Core v30
@ 2025-04-28 11:34 Sjors Provoost
2025-04-28 16:25 ` Sjors Provoost
0 siblings, 1 reply; 3+ messages in thread
From: Sjors Provoost @ 2025-04-28 11:34 UTC (permalink / raw)
To: Bitcoin Development Mailing List
Dear list,
I thought this was already communicated, but I can't find a thread.
A recent pull request was merged into Bitcoin Core that removes existing checkpoints and all related support code.[0] The current plan is for this change to be part of v30 which should be released this fall.
Checkpoints are no longer needed in order to protect against low proof-of-work header spam attacks, thanks to the headers pre-sync functionality included since v24. [1]
The existing checkpoints no longer provide enough security against those attacks. Adding new checkpoints is not desirable.
However, if someone is aware of another attack, please disclose it (ideally responsibly) so the change can be reverted. [2]
Since older versions of Bitcoin Core, as well as other Bitcoin implementations, still enforce the checkpoints, an extremely large reorg (attack by aliens, NSA, etc) would split the network. This may already be the case due to the way some older soft forks have been grandfathered / buried in Bitcoin Core.[3]
I think many would consider Bitcoin a total loss in such a scenario and not worth salvaging - especially if the attacker didn't airdrop you any coins in the reorg. But it could be useful for someone to write a BIP that describes what happens in such a scenario. It could recommend ways to at least regain consistency, and perhaps permanently apply some soft fork rules from genesis (e.g. activate them if the last checkpoint header is absent).
Kind regards,
Sjors Provoost
[0] https://github.com/bitcoin/bitcoin/pull/31649
[1] https://bitcoin.stackexchange.com/a/121235/4948
[2] https://github.com/bitcoin/bitcoin/blob/master/SECURITY.md
[3] https://github.com/bitcoin/bips/blob/master/bip-0090.mediawiki
--
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/F8E9B25A-5198-4A5E-B3D7-9DAD6B709825%40sprovoost.nl.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [bitcoindev] Removing checkpoints in Bitcoin Core v30
2025-04-28 11:34 [bitcoindev] Removing checkpoints in Bitcoin Core v30 Sjors Provoost
@ 2025-04-28 16:25 ` Sjors Provoost
2025-05-02 21:16 ` eric
0 siblings, 1 reply; 3+ messages in thread
From: Sjors Provoost @ 2025-04-28 16:25 UTC (permalink / raw)
To: Bitcoin Development Mailing List
Note that my original mail was written in a hurry and does not provide a full overview of the arguments for and against checkpoints. Perhaps this thread can serve as a collection point for the various historical discussions that were had on this topic.
In the context of BIP30 [0] Eric Voskuil brought up performance:
> The top checkpoint is consensus for over 11 years and materially reduces the validation cost of 295,000 blocks.
I don't think performance should be a consideration when removing checkpoints.
Afaik checkpoints were not introduced as a performance feature, but rather as a DoS vulnerability fix.
Even if they were, imo they shouldn't be used for performance enhancement, as it creates a temptation to add more - although Eric clearly says:
> I would never advocate for adding more
But perhaps someone else would fork Bitcoin Core (or Libbitcoin) and start with honest checkpoints, in order to gain adoption due its excellent performance - especially when combined with a UTXO snapshot. And then suddenly it has an "upgrade".
But even if this wasn't a concern, the performance benefit of the latest checkpoint from 2014 is just very small. That's because the early blocks don't have that many signatures to check. The first 5 years of Bitcoin represent only 3% of all historical transactions.
On an M4 MacBook I did an IBD up to the last checkpoint at height 295,000. This took 17.5 minutes.
I then ran it again with -assumevalid=0, i.e. checking all signatures, which took 18 minutes.
These times are from connecting the first block until connecting block 295,000. They exclude the 1.5 minutes spent on the header pre-sync mechanism, which is required to operate safely with or without (new) checkpoints. In both cases I used -dbcache=3000 to avoid UTXO set disk I/O.
The difference is 30 seconds, but let's round that up to 1 minute because my setup isn't a clean lab.
A full IBD takes at least 5 hours on this machine, so the benefit is roughly 0.3% (unless we add another checkpoint).
Loading a UTXO snapshot at the checkpoint height, without doing background validation of historical blocks, might get us an additional 15 minute speedup, which is still only 5%.
Libbitcoin would probably find a different ratio, as might a machine with severe CPU limitations, but I suspect it will be equally negliable. And without new checkpoints the ratio keeps going down over time.
Note that -assumevalid doesn't need checkpoints, it just falls back to more slow validation if there is a major reorg (but perhaps sub catastrophic). Its performance benefit for the present day blockchain should be much higher than what I found here.
That said, I've never been a fan of -assumevalid conceptually. Hopefully it can be replaced by something like AssumeUTXO + SwiftSync [1] for the background with full validation using block Undo data.
- Sjors
[0] https://gnusha.org/pi/bitcoindev/E8225EAC-BED8-4840-8E3D-81A55C365209@voskuil.org/T/#mf3d290b70bae822cf31a683439ca8dbaed443e42
[1] https://gnusha.org/pi/bitcoindev/CAPv7TjaM0tfbcBTRa0_713Bk6Y9jr+ShOC1KZi2V3V2zooTXyg@mail.gmail.com/
--
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/8E4DFC2E-23D4-4D22-87C5-415A3CFC7E57%40sprovoost.nl.
^ permalink raw reply [flat|nested] 3+ messages in thread
* RE: [bitcoindev] Removing checkpoints in Bitcoin Core v30
2025-04-28 16:25 ` Sjors Provoost
@ 2025-05-02 21:16 ` eric
0 siblings, 0 replies; 3+ messages in thread
From: eric @ 2025-05-02 21:16 UTC (permalink / raw)
To: 'Sjors Provoost', 'Bitcoin Development Mailing List'
Hi Sjors,
> In the context of BIP30 [0] Eric Voskuil brought up performance:
I didn't originate the point on performance. I said (in the BIP30 thread):
> One reference states that “assume valid” speeds IBD, but of course it does so by not validating.
Which was in response to your OP of this thread, which provided that reference, specifically:
"Assumed Valid Blocks: a feature designed to replace the secondary use of checkpoints for (optionally) speeding up Initial Block Download (IBD) by skipping validation of signatures in old blocks. This was deployed in Bitcoin Core 0.14" - David A. Harding
This in turn references the following discussion:
"Bitcoin 0.5.0 built upon those checkpoints to speed up IBD by skipping verification of signatures in blocks that were earlier in the block chain than the most recent checkpoint."
https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> > The top checkpoint is consensus for over 11 years and materially reduces
> the validation cost of 295,000 blocks.
>
> I don't think performance should be a consideration when removing
> checkpoints.
To be perfectly clear, I am not arguing against this hard fork because it reduces IBD cost. I’m pointing out that one of the arguments *in favor* of removing checkpoints is that "assume valid" now serves this "secondary use". However, as I pointed out, assume valid is not consensus, it achieves this outcome by trusting, not validating. The existing checkpoints are consensus - they provide this advantage when fully validating.
I'm not suggesting that checkpoints need to be added to improve performance. I'm pointing out that removing them hurts performance. So performance is obviously not a reason to accept such a hard fork.
> Afaik checkpoints were not introduced as a performance feature, but rather as
> a DoS vulnerability fix.
It appears from the above discussion that there was more than one reason. However this isn't relevant. What matters is the consequences of removing them, not why they were introduced.
> Even if they were, imo they shouldn't be used for performance enhancement,
> as it creates a temptation to add more - although Eric clearly says:
>
> > I would never advocate for adding more
>
> But perhaps someone else would fork Bitcoin Core (or Libbitcoin) and start
> with honest checkpoints, in order to gain adoption due its excellent
> performance - especially when combined with a UTXO snapshot. And then
> suddenly it has an "upgrade".
>
> But even if this wasn't a concern,
This feels more like speculation than a reasonable concern. To date bitcoind has introduced two trust-based mechanisms (assume valid and assume utxo) both with the objective of avoiding validation to improve IBD time. As I understand it, BC ships with a very recent assume valid configuration by default. If you have this concern, it should be directed these current practices, not at ancient checkpoints. Removing the latter does not eliminate the former - which should be the current concern. I've recently seen declarations that the IBD problem is "solved" because of assume valid and/or assume utxo. That's not speculation.
> the performance benefit of the latest
> checkpoint from 2014 is just very small. That's because the early blocks don't
> have that many signatures to check. The first 5 years of Bitcoin represent only
> 3% of all historical transactions.
This assumes that signature validation is the major cost, which is not the case for bitcoind architectures.
> On an M4 MacBook I did an IBD up to the last checkpoint at height 295,000.
> This took 17.5 minutes.
I have done this many times, on powerful machines, and it's generally at least an hour - even with checkpoints (and headers excluded).
> I then ran it again with -assumevalid=0, i.e. checking all signatures, which took
> 18 minutes.
The major cost for the bitcoind architecture is unspent output accumulation. There is only a modest distinction between full validation and assume valid across the entire chain. As a result this measurement doesn't reflect the true (necessary) cost differential. I just performed a 1-295,000 block libbitcoin sync with and without checkpoints (and no milestone/assume valid). The checkpointed time is 3.4 minutes and without is 8.3 minutes (on the same machine BC takes 69.3 minutes with checkpoints). So checkpoints validated at 41% of the time vs. without (vs. your 97%). Again, this is not the basis of my comments, just clarifying for the record.
> These times are from connecting the first block until connecting block
> 295,000. They exclude the 1.5 minutes spent on the header pre-sync
> mechanism, which is required to operate safely with or without (new)
> checkpoints.
Same in both cases.
> Libbitcoin would probably find a different ratio, as might a machine with
> severe CPU limitations, but I suspect it will be equally negliable. And without
> new checkpoints the ratio keeps going down over time.
Of course, the ratio is small and will keep decreasing. On this machine it's a difference of 4.9 minutes out of 4.4 hours for libbitcoin. The ratio is smaller for BC, as it takes 24.5 hours to sync to the same height even with assume valid.
> That said, I've never been a fan of -assumevalid conceptually.
I agree, not a fan. We provide milestones (same security model) so that people can rapidly reproduce a previously synced node from the network by specifying a milestone that they have previously validated. On my current low end ($350) benchmark on a 2.3 Gbps Internet, this takes under 40 minutes to block 850k, and checkpoints don't make a difference either way. It can be faster to sync from the p2p network than to copy the store, and of course doesn't require a local copy. So it's a useful feature, but not a substitute for validation.
> Hopefully it can
> be replaced by something like AssumeUTXO + SwiftSync [1] for the
> background with full validation using block Undo data.
I'm not so clear on why BC wants to give people the impression that a node is usable when its chain hasn't been validated. I don't see this as the "hopeful" outcome, but that's another topic I suppose.
I also don't see the reason for all of this complexity to end up with a security model identical to assume valid (until it's actually validated). Why not just sync the strong blocks and validate them in the background without all of the fuss, extra bandwidth, and precomputations?
In any case, I still do not see ANY justification for hard forking out the checkpoints, and there are of course legitimate reasons to avoid doing so.
https://groups.google.com/g/bitcoindev/c/aqHRfa0UWFo
Best,
Eric
--
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/035701dbbba7%24807f23b0%24817d6b10%24%40voskuil.org.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2025-05-02 22:24 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-04-28 11:34 [bitcoindev] Removing checkpoints in Bitcoin Core v30 Sjors Provoost
2025-04-28 16:25 ` Sjors Provoost
2025-05-02 21:16 ` eric
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox