* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-11 23:34 [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core Pieter Wuille
@ 2014-10-12 5:51 ` Aaron Voisine
2014-10-12 7:14 ` Gregory Maxwell
` (3 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Aaron Voisine @ 2014-10-12 5:51 UTC (permalink / raw)
To: Pieter Wuille; +Cc: Bitcoin Dev
This is great Pieter. I was able to sync the entire blockchain from
scratch in a little over 4 hours on a laptop over cable modem. :) No
issues to report. Even my family photos are intact! This makes it
practical to run a full node, part time on a laptop again.
Aaron Voisine
breadwallet.com
On Sat, Oct 11, 2014 at 4:34 PM, Pieter Wuille <pieter.wuille@gmail•com> wrote:
> Hi all,
>
> I believe that a large change that I've been working on for Bitcoin
> Core is ready for review and testing: headers-first synchronization.
> In short, it changes the way the best chain is discovered, downloaded
> and verified, with several advantages:
> * Parallel block downloading (much faster sync on typical network connections).
> * No more stalled downloads.
> * Much more robust against unresponsive or slow peers.
> * Removes a class of DoS attacks related to peers feeding you
> low-difficulty valid large blocks on a side branch.
> * Reduces the need for checkpoints in the code.
> * No orphan blocks stored in memory anymore (reducing memory usage during sync).
> * A major step step towards an SPV mode using the reference codebase.
>
> Historically, this mode of operation has been known for years (Greg
> Maxwell wrote up a description of a very similar method in
> https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync
> in early 2012, but it was known before that), but it took a long time
> to refactor these code enough to support it.
>
> Technically, it works by replacing the single-peer blocks download by
> a single-peer headers download (which typically takes seconds/minutes)
> and verification, and simultaneously fetching blocks along the best
> known headers chain from all peers that are known to have the relevant
> blocks. Downloading is constrained to a moving window to avoid
> unbounded unordering of blocks on disk (which would interfere with
> pruning later).
>
> At the protocol level, it increases the minimally supported version
> for peers to 31800 (corresponding to bitcoin v3.18, released in
> december 2010), as earlier versions did not support the getheaders P2P
> message.
>
> So, the code is available as a github pull request
> (https://github.com/bitcoin/bitcoin/pull/4468), or packaged on
> http://bitcoin.sipa.be/builds/headersfirst, where you can also find
> binaries to test with.
>
> Known issues:
> * At the very start of the sync, especially before all headers are
> processed, downloading is very slow due to a limited number of blocks
> that are requested per peer simultaneously. The policies around this
> will need some experimentation can certainly be improved.
> * Blocks will be stored on disk out of order (in the order they are
> received, really), which makes it incompatible with some tools or
> other programs. Reindexing using earlier versions will also not work
> anymore as a result of this.
> * The block index database will now hold headers for which no block is
> stored on disk, which earlier versions won't support. If you are fully
> synced, it may still be possible to go back to an earlier version.
>
> Unknown issues:
> * Who knows, maybe it will replace your familiy pictures with Nyan
> Cat? Use at your own risk.
>
> TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or
> http://bitcoin.sipa.be/builds/headersfirst.
>
> --
> Pieter
>
> ------------------------------------------------------------------------------
> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-11 23:34 [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core Pieter Wuille
2014-10-12 5:51 ` Aaron Voisine
@ 2014-10-12 7:14 ` Gregory Maxwell
2014-10-12 8:41 ` Geir Harald Hansen
` (2 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Gregory Maxwell @ 2014-10-12 7:14 UTC (permalink / raw)
To: Pieter Wuille; +Cc: Bitcoin Dev
On Sat, Oct 11, 2014 at 11:34 PM, Pieter Wuille <pieter.wuille@gmail•com> wrote:
> * Parallel block downloading (much faster sync on typical network connections).
"Much faster" is an understatement. Benchmarking here shows one hour
five minutes syncing to 295000. Old code isn't even at 250000 after
7 hours.
(I'm using 295k as the target here because after that point ecdsa
dominates, and then your 6+x faster libsecp256k1 makes more of a
difference)
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-11 23:34 [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core Pieter Wuille
2014-10-12 5:51 ` Aaron Voisine
2014-10-12 7:14 ` Gregory Maxwell
@ 2014-10-12 8:41 ` Geir Harald Hansen
2014-10-12 9:44 ` Luke Dashjr
2014-10-12 9:52 ` Wladimir
2014-10-12 19:13 ` Jameson Lopp
2014-10-16 5:05 ` Rebroad (sourceforge)
4 siblings, 2 replies; 10+ messages in thread
From: Geir Harald Hansen @ 2014-10-12 8:41 UTC (permalink / raw)
To: bitcoin-development
On 12.10.2014 01:34, Pieter Wuille wrote:
> * No orphan blocks stored in memory anymore (reducing memory usage during sync).
Will this slow down reorgs after a fork, compared to today?
Regards,
Geir H. Hansen, Bitminter
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-12 8:41 ` Geir Harald Hansen
@ 2014-10-12 9:44 ` Luke Dashjr
2014-10-12 9:52 ` Wladimir
1 sibling, 0 replies; 10+ messages in thread
From: Luke Dashjr @ 2014-10-12 9:44 UTC (permalink / raw)
To: bitcoin-development
On Sunday, October 12, 2014 8:41:29 AM Geir Harald Hansen wrote:
> On 12.10.2014 01:34, Pieter Wuille wrote:
> > * No orphan blocks stored in memory anymore (reducing memory usage during
> > sync).
>
> Will this slow down reorgs after a fork, compared to today?
It shouldn't... he's talking about actual orphan blocks (ones without a known
previous/parent block), not stale blocks.
Luke
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-12 8:41 ` Geir Harald Hansen
2014-10-12 9:44 ` Luke Dashjr
@ 2014-10-12 9:52 ` Wladimir
2014-10-12 10:06 ` Geir Harald Hansen
1 sibling, 1 reply; 10+ messages in thread
From: Wladimir @ 2014-10-12 9:52 UTC (permalink / raw)
To: Geir Harald Hansen; +Cc: Bitcoin Dev
On Sun, Oct 12, 2014 at 10:41 AM, Geir Harald Hansen
<operator@bitminter•com> wrote:
> On 12.10.2014 01:34, Pieter Wuille wrote:
>> * No orphan blocks stored in memory anymore (reducing memory usage during sync).
>
> Will this slow down reorgs after a fork, compared to today?
Why would you think so? Orphan blocks are blocks whose parent is not
known. In the case of a reorganization the client 'jumps' to a new
best chain, for this to happen the original tip and the new best tip
and all their parents must be already known.
Wladimir
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-12 9:52 ` Wladimir
@ 2014-10-12 10:06 ` Geir Harald Hansen
0 siblings, 0 replies; 10+ messages in thread
From: Geir Harald Hansen @ 2014-10-12 10:06 UTC (permalink / raw)
To: Bitcoin Dev
On 12.10.2014 11:52, Wladimir wrote:
> On Sun, Oct 12, 2014 at 10:41 AM, Geir Harald Hansen
> <operator@bitminter•com> wrote:
>> On 12.10.2014 01:34, Pieter Wuille wrote:
>>> * No orphan blocks stored in memory anymore (reducing memory usage during sync).
>>
>> Will this slow down reorgs after a fork, compared to today?
>
> Why would you think so? Orphan blocks are blocks whose parent is not
> known. In the case of a reorganization the client 'jumps' to a new
> best chain, for this to happen the original tip and the new best tip
> and all their parents must be already known.
I was thinking of shorter sides of forks as being orphaned. Sorry for
the confusion.
Regards,
Geir H. Hansen, Bitminter
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-11 23:34 [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core Pieter Wuille
` (2 preceding siblings ...)
2014-10-12 8:41 ` Geir Harald Hansen
@ 2014-10-12 19:13 ` Jameson Lopp
2014-10-13 22:43 ` 21E14
2014-10-16 5:05 ` Rebroad (sourceforge)
4 siblings, 1 reply; 10+ messages in thread
From: Jameson Lopp @ 2014-10-12 19:13 UTC (permalink / raw)
To: bitcoin-development
Great work, Pieter. I've been spooling up several nodes per week lately and can testify that stalled downloads during initial syncing are a pain. I usually forgo bootstrapping on VPSes because I don't want to have to adjust the disk space allocation.
With headers-first I'm saturating my home cable connection with download rates of 4 MB/s until block 295,000 at which point CPU becomes the bottleneck and it settles down in the 1 MB/s range.
It took 6 minutes for my node to sync to block height 100,000
22 minutes to reach height 200,000
62 minutes to reach height 250,000
125 minutes to reach height 295,000
144 minutes to reach height 300,000
248 minutes to reach height 325,000
- Jameson
On 10/11/2014 07:34 PM, Pieter Wuille wrote:
> Hi all,
>
> I believe that a large change that I've been working on for Bitcoin
> Core is ready for review and testing: headers-first synchronization.
> In short, it changes the way the best chain is discovered, downloaded
> and verified, with several advantages:
> * Parallel block downloading (much faster sync on typical network connections).
> * No more stalled downloads.
> * Much more robust against unresponsive or slow peers.
> * Removes a class of DoS attacks related to peers feeding you
> low-difficulty valid large blocks on a side branch.
> * Reduces the need for checkpoints in the code.
> * No orphan blocks stored in memory anymore (reducing memory usage during sync).
> * A major step step towards an SPV mode using the reference codebase.
>
> Historically, this mode of operation has been known for years (Greg
> Maxwell wrote up a description of a very similar method in
> https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync
> in early 2012, but it was known before that), but it took a long time
> to refactor these code enough to support it.
>
> Technically, it works by replacing the single-peer blocks download by
> a single-peer headers download (which typically takes seconds/minutes)
> and verification, and simultaneously fetching blocks along the best
> known headers chain from all peers that are known to have the relevant
> blocks. Downloading is constrained to a moving window to avoid
> unbounded unordering of blocks on disk (which would interfere with
> pruning later).
>
> At the protocol level, it increases the minimally supported version
> for peers to 31800 (corresponding to bitcoin v3.18, released in
> december 2010), as earlier versions did not support the getheaders P2P
> message.
>
> So, the code is available as a github pull request
> (https://github.com/bitcoin/bitcoin/pull/4468), or packaged on
> http://bitcoin.sipa.be/builds/headersfirst, where you can also find
> binaries to test with.
>
> Known issues:
> * At the very start of the sync, especially before all headers are
> processed, downloading is very slow due to a limited number of blocks
> that are requested per peer simultaneously. The policies around this
> will need some experimentation can certainly be improved.
> * Blocks will be stored on disk out of order (in the order they are
> received, really), which makes it incompatible with some tools or
> other programs. Reindexing using earlier versions will also not work
> anymore as a result of this.
> * The block index database will now hold headers for which no block is
> stored on disk, which earlier versions won't support. If you are fully
> synced, it may still be possible to go back to an earlier version.
>
> Unknown issues:
> * Who knows, maybe it will replace your familiy pictures with Nyan
> Cat? Use at your own risk.
>
> TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or
> http://bitcoin.sipa.be/builds/headersfirst.
>
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-12 19:13 ` Jameson Lopp
@ 2014-10-13 22:43 ` 21E14
0 siblings, 0 replies; 10+ messages in thread
From: 21E14 @ 2014-10-13 22:43 UTC (permalink / raw)
To: Jameson Lopp; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 5240 bytes --]
When forgoing bootstrapping due to disk space constraints, you, and the
network, are likely better off -reindex-ing from current blk000??.dat files.
Which brings up an interesting point: The improvements related to the
headers first approach are likely to increase, how ever marginally, the
percentage of block exchange-related traffic, as it is less painful now to
be catching up.
It'd be interesting to see the statistics, not from a single node
perspective, but from the viewpoint of an Internet backbone provider, say
through the cables coming ashore in Cornwall. For the incurred bandwidth
expense would invariably trickle down to transaction fees in an equilibrium
model.
There is an opportunity somewhere in this.
On Sun, Oct 12, 2014 at 7:13 PM, Jameson Lopp <jameson.lopp@gmail•com>
wrote:
> Great work, Pieter. I've been spooling up several nodes per week lately
> and can testify that stalled downloads during initial syncing are a pain. I
> usually forgo bootstrapping on VPSes because I don't want to have to adjust
> the disk space allocation.
>
> With headers-first I'm saturating my home cable connection with download
> rates of 4 MB/s until block 295,000 at which point CPU becomes the
> bottleneck and it settles down in the 1 MB/s range.
>
> It took 6 minutes for my node to sync to block height 100,000
> 22 minutes to reach height 200,000
> 62 minutes to reach height 250,000
> 125 minutes to reach height 295,000
> 144 minutes to reach height 300,000
> 248 minutes to reach height 325,000
>
> - Jameson
>
> On 10/11/2014 07:34 PM, Pieter Wuille wrote:
> > Hi all,
> >
> > I believe that a large change that I've been working on for Bitcoin
> > Core is ready for review and testing: headers-first synchronization.
> > In short, it changes the way the best chain is discovered, downloaded
> > and verified, with several advantages:
> > * Parallel block downloading (much faster sync on typical network
> connections).
> > * No more stalled downloads.
> > * Much more robust against unresponsive or slow peers.
> > * Removes a class of DoS attacks related to peers feeding you
> > low-difficulty valid large blocks on a side branch.
> > * Reduces the need for checkpoints in the code.
> > * No orphan blocks stored in memory anymore (reducing memory usage
> during sync).
> > * A major step step towards an SPV mode using the reference codebase.
> >
> > Historically, this mode of operation has been known for years (Greg
> > Maxwell wrote up a description of a very similar method in
> > https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync
> > in early 2012, but it was known before that), but it took a long time
> > to refactor these code enough to support it.
> >
> > Technically, it works by replacing the single-peer blocks download by
> > a single-peer headers download (which typically takes seconds/minutes)
> > and verification, and simultaneously fetching blocks along the best
> > known headers chain from all peers that are known to have the relevant
> > blocks. Downloading is constrained to a moving window to avoid
> > unbounded unordering of blocks on disk (which would interfere with
> > pruning later).
> >
> > At the protocol level, it increases the minimally supported version
> > for peers to 31800 (corresponding to bitcoin v3.18, released in
> > december 2010), as earlier versions did not support the getheaders P2P
> > message.
> >
> > So, the code is available as a github pull request
> > (https://github.com/bitcoin/bitcoin/pull/4468), or packaged on
> > http://bitcoin.sipa.be/builds/headersfirst, where you can also find
> > binaries to test with.
> >
> > Known issues:
> > * At the very start of the sync, especially before all headers are
> > processed, downloading is very slow due to a limited number of blocks
> > that are requested per peer simultaneously. The policies around this
> > will need some experimentation can certainly be improved.
> > * Blocks will be stored on disk out of order (in the order they are
> > received, really), which makes it incompatible with some tools or
> > other programs. Reindexing using earlier versions will also not work
> > anymore as a result of this.
> > * The block index database will now hold headers for which no block is
> > stored on disk, which earlier versions won't support. If you are fully
> > synced, it may still be possible to go back to an earlier version.
> >
> > Unknown issues:
> > * Who knows, maybe it will replace your familiy pictures with Nyan
> > Cat? Use at your own risk.
> >
> > TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or
> > http://bitcoin.sipa.be/builds/headersfirst.
> >
>
>
> ------------------------------------------------------------------------------
> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 6765 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core
2014-10-11 23:34 [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core Pieter Wuille
` (3 preceding siblings ...)
2014-10-12 19:13 ` Jameson Lopp
@ 2014-10-16 5:05 ` Rebroad (sourceforge)
4 siblings, 0 replies; 10+ messages in thread
From: Rebroad (sourceforge) @ 2014-10-16 5:05 UTC (permalink / raw)
Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 5319 bytes --]
Hi all,
I've also been spending a few months coding upon the change's Pieter has
been making with the headersfirst8 pull request.
My code updates are also ready to test, and are available on github at
https://github.com/rebroad/bitcoin/ and the branch is
"sipa-headersfirst8-patches".
I've made a number of improvement. Namely that it tracks the block as it
downloads and won't disconnect if the block is downloading at a reasonable
speed. The current stall logic of Pieter's is broken in that it will
continue to disconnect a peer that is providing a block - particularly the
next block needed to advance the current tip. I've raised this issue, but
so far haven't been able to communicate the problem in a way that's been
understood.
I've also added logic to avoid the node stalling due to many blocks being
added to the ActiveTip (which would cause timeouts both from our node, and
nodes we are connected to). It will also log and determine bandwidth per
node, and the next changes I will be adding will be to make it prefer to
download from the faster nodes (coming shortly).
I have also added code ready to adapt the window size for the download.
Currently the start setting for blocks in flight is 3 per node, but for
early on on the block chain this will be too small, so once it realises
this after a few downloads and determines the average block size and speed,
it will automatically adjust the number of blocks to request per node and
revise this each minute.
Please do take a look at my code, and feel free to test it. It also
improves some of the debug.log output to make it easier to read and provide
useful information about concurrent downloads, etc.
Edmund
On Sun, Oct 12, 2014 at 7:34 AM, Pieter Wuille <pieter.wuille@gmail•com>
wrote:
> Hi all,
>
> I believe that a large change that I've been working on for Bitcoin
> Core is ready for review and testing: headers-first synchronization.
> In short, it changes the way the best chain is discovered, downloaded
> and verified, with several advantages:
> * Parallel block downloading (much faster sync on typical network
> connections).
> * No more stalled downloads.
> * Much more robust against unresponsive or slow peers.
> * Removes a class of DoS attacks related to peers feeding you
> low-difficulty valid large blocks on a side branch.
> * Reduces the need for checkpoints in the code.
> * No orphan blocks stored in memory anymore (reducing memory usage during
> sync).
> * A major step step towards an SPV mode using the reference codebase.
>
> Historically, this mode of operation has been known for years (Greg
> Maxwell wrote up a description of a very similar method in
> https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync
> in early 2012, but it was known before that), but it took a long time
> to refactor these code enough to support it.
>
> Technically, it works by replacing the single-peer blocks download by
> a single-peer headers download (which typically takes seconds/minutes)
> and verification, and simultaneously fetching blocks along the best
> known headers chain from all peers that are known to have the relevant
> blocks. Downloading is constrained to a moving window to avoid
> unbounded unordering of blocks on disk (which would interfere with
> pruning later).
>
> At the protocol level, it increases the minimally supported version
> for peers to 31800 (corresponding to bitcoin v3.18, released in
> december 2010), as earlier versions did not support the getheaders P2P
> message.
>
> So, the code is available as a github pull request
> (https://github.com/bitcoin/bitcoin/pull/4468), or packaged on
> http://bitcoin.sipa.be/builds/headersfirst, where you can also find
> binaries to test with.
>
> Known issues:
> * At the very start of the sync, especially before all headers are
> processed, downloading is very slow due to a limited number of blocks
> that are requested per peer simultaneously. The policies around this
> will need some experimentation can certainly be improved.
> * Blocks will be stored on disk out of order (in the order they are
> received, really), which makes it incompatible with some tools or
> other programs. Reindexing using earlier versions will also not work
> anymore as a result of this.
> * The block index database will now hold headers for which no block is
> stored on disk, which earlier versions won't support. If you are fully
> synced, it may still be possible to go back to an earlier version.
>
> Unknown issues:
> * Who knows, maybe it will replace your familiy pictures with Nyan
> Cat? Use at your own risk.
>
> TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or
> http://bitcoin.sipa.be/builds/headersfirst.
>
> --
> Pieter
>
>
> ------------------------------------------------------------------------------
> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 6718 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread