* Re: [Bitcoin-development] Service bits for pruned nodes
@ 2013-05-16 11:26 Ricardo Filipe
2013-05-16 15:47 ` Jeff Garzik
0 siblings, 1 reply; 26+ messages in thread
From: Ricardo Filipe @ 2013-05-16 11:26 UTC (permalink / raw)
To: bitcoin-development
We would only end up with few copies of the historic data if users
could choose what parts of the blockchain to store. Simply store
chunks randomly, according to users available space, and give priority
to the "N most recent" chunks to have more replicas in the network.
You don't need bittorrent specifically for a DHT, if publicity is a
problem. There are many DHT proposals and implementations, and i bet
one of them should be more suitable to the bitcoin network than
bittorrent's.
>On Sun, Apr 28, 2013 at 9:29 AM, Mike Hearn <mike@...> wrote:
>> I'd imagined that nodes would be able to pick their own ranges to keep
>> rather than have fixed chosen intervals. "Everything or two weeks" is rather
>
>X most recent is special for two reasons: It meshes well with actual demand,
>and the data is required for reorganization.
>
>So whatever we do for historic data, N most recent should be treated specially.
>
>But I also agree that its important that <everything> be splittable into ranges
>because otherwise when having to choose between serving historic data
>and— say— 40 GB storage, a great many are going to choose not to serve
>historic data... and so nodes may be willing to contribute 4-39 GB storage
>to the network there will be no good way for them to do so and we may end
>up with too few copies of the historic data available.
>
>As can be seen in the graph, once you get past the most recent 4000
>blocks the probability is fairly uniform... so "N most recent" is not a
>good way to divide load for the older blocks. But simple ranges— perhaps
>quantized to groups of 100 or 1000 blocks or something— would work fine.
>
>This doesn't have to come in the first cut, however— and it needs new
>addr messages in any case.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-16 11:26 [Bitcoin-development] Service bits for pruned nodes Ricardo Filipe
@ 2013-05-16 15:47 ` Jeff Garzik
2013-05-16 16:23 ` Ricardo Filipe
0 siblings, 1 reply; 26+ messages in thread
From: Jeff Garzik @ 2013-05-16 15:47 UTC (permalink / raw)
To: Ricardo Filipe; +Cc: bitcoin-development
On Thu, May 16, 2013 at 7:26 AM, Ricardo Filipe
<ricardojdfilipe@gmail•com> wrote:
> We would only end up with few copies of the historic data if users
> could choose what parts of the blockchain to store. Simply store
> chunks randomly, according to users available space, and give priority
> to the "N most recent" chunks to have more replicas in the network.
>
> You don't need bittorrent specifically for a DHT, if publicity is a
> problem. There are many DHT proposals and implementations, and i bet
> one of them should be more suitable to the bitcoin network than
> bittorrent's.
That's just about the worst thing you could do for bitcoin. DoS one
part of the DHT, you DoS the entire blockchain by breaking the chain.
--
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti•com
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-16 15:47 ` Jeff Garzik
@ 2013-05-16 16:23 ` Ricardo Filipe
0 siblings, 0 replies; 26+ messages in thread
From: Ricardo Filipe @ 2013-05-16 16:23 UTC (permalink / raw)
To: Jeff Garzik; +Cc: bitcoin-development
Of course.
My proposal was just for the pruned nodes.
I.e. You would have a majority (maybe not even a majority required) of
nodes storing the whole blockchain and pruned nodes would store
"random" parts of the blockchain, according to the resources they
have, which would be organized as a DHT.
2013/5/16 Jeff Garzik <jgarzik@exmulti•com>:
> On Thu, May 16, 2013 at 7:26 AM, Ricardo Filipe
> <ricardojdfilipe@gmail•com> wrote:
>> We would only end up with few copies of the historic data if users
>> could choose what parts of the blockchain to store. Simply store
>> chunks randomly, according to users available space, and give priority
>> to the "N most recent" chunks to have more replicas in the network.
>>
>> You don't need bittorrent specifically for a DHT, if publicity is a
>> problem. There are many DHT proposals and implementations, and i bet
>> one of them should be more suitable to the bitcoin network than
>> bittorrent's.
>
> That's just about the worst thing you could do for bitcoin. DoS one
> part of the DHT, you DoS the entire blockchain by breaking the chain.
>
> --
> Jeff Garzik
> exMULTI, Inc.
> jgarzik@exmulti•com
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-06 8:19 ` Mike Hearn
@ 2013-05-06 13:13 ` Pieter Wuille
0 siblings, 0 replies; 26+ messages in thread
From: Pieter Wuille @ 2013-05-06 13:13 UTC (permalink / raw)
To: Mike Hearn; +Cc: Bitcoin Dev
On Mon, May 06, 2013 at 10:19:35AM +0200, Mike Hearn wrote:
> You are welcome to optimise P2P addr broadcasts or develop better bootstrap
> mechanisms.
I think John's actually has a point here. If we're judging the quality of a
protocol change by how compatible it is with DNS seeding, then we're clearly not
using DNS seeding as seeding anymore (=getting an entry point into the P2P
network), but as a mechanism for choosing (all) peers.
Eventually, I think it makes sense to move to a system where you get seeds from
a DNS (or other mechanism), connect to one or a few of the results, do a getaddr,
fill your peer IP database with it, and disconnect from the DNS seeded peer.
This probably means we need to look at ways to optimize current peer exchange,
but that's certainly welcome in any case.
--
Pieter
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-05 13:12 ` John Dillon
@ 2013-05-06 8:19 ` Mike Hearn
2013-05-06 13:13 ` Pieter Wuille
0 siblings, 1 reply; 26+ messages in thread
From: Mike Hearn @ 2013-05-06 8:19 UTC (permalink / raw)
To: John Dillon; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]
You are welcome to optimise P2P addr broadcasts or develop better bootstrap
mechanisms.
On Sun, May 5, 2013 at 3:12 PM, John Dillon
<john.dillon892@googlemail•com>wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Sorry I should have used the word bootstrapping there rather than
> discovery.
> But again I think that shows my point clearly. Centralized methods like DNS
> should be used for as little as possible, just simple initial
> bootstrapping,
> and focus the development efforts towards the non-centralized peer
> discovery
> mechanisms.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQEcBAEBCAAGBQJRhlpyAAoJEEWCsU4mNhiP+NwH/3RY5vBpSYkwKgTmdKHRc/gw
> BJCSV/1MEDECgBTxaRYSzYZyargjsdG50KaIaCq8M1+8DWkBEkH8JFif7UYMlZGM
> WROMP6UjAnP1fJ3B2JChdMgRv1HdXJQDQVcO8UnSJschhX8lZZiUySbaqIPuRuV/
> lI7/JkUZvmnms4+HGiaqwfbPO0k6ytJNKxORrk4TzFnThh4dy9WytElc8JHZOFaQ
> ly159X5JuEwh8DLOoUtPhaR6tJaJbJLBEt+QJiGnSktPsJCE8p9+4HQ0kMCQr3Ha
> 05EHTZEw+TqEPaA7vFLgA/9tWjK9s1Y6sqLOAYiLp/0wSKzCkBO0C5LWFHsJ/XQ=
> =aCgi
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite
> It's a free troubleshooting tool designed for production
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://p.sf.net/sfu/appdyn_d2d_ap2
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 2363 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-04 18:55 ` Jeff Garzik
@ 2013-05-05 13:12 ` John Dillon
2013-05-06 8:19 ` Mike Hearn
0 siblings, 1 reply; 26+ messages in thread
From: John Dillon @ 2013-05-05 13:12 UTC (permalink / raw)
To: Jeff Garzik; +Cc: Bitcoin Dev
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Sorry I should have used the word bootstrapping there rather than discovery.
But again I think that shows my point clearly. Centralized methods like DNS
should be used for as little as possible, just simple initial bootstrapping,
and focus the development efforts towards the non-centralized peer discovery
mechanisms.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBCAAGBQJRhlpyAAoJEEWCsU4mNhiP+NwH/3RY5vBpSYkwKgTmdKHRc/gw
BJCSV/1MEDECgBTxaRYSzYZyargjsdG50KaIaCq8M1+8DWkBEkH8JFif7UYMlZGM
WROMP6UjAnP1fJ3B2JChdMgRv1HdXJQDQVcO8UnSJschhX8lZZiUySbaqIPuRuV/
lI7/JkUZvmnms4+HGiaqwfbPO0k6ytJNKxORrk4TzFnThh4dy9WytElc8JHZOFaQ
ly159X5JuEwh8DLOoUtPhaR6tJaJbJLBEt+QJiGnSktPsJCE8p9+4HQ0kMCQr3Ha
05EHTZEw+TqEPaA7vFLgA/9tWjK9s1Y6sqLOAYiLp/0wSKzCkBO0C5LWFHsJ/XQ=
=aCgi
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-04 18:07 ` John Dillon
@ 2013-05-04 18:55 ` Jeff Garzik
2013-05-05 13:12 ` John Dillon
0 siblings, 1 reply; 26+ messages in thread
From: Jeff Garzik @ 2013-05-04 18:55 UTC (permalink / raw)
To: John Dillon; +Cc: Bitcoin Dev
On Sat, May 4, 2013 at 2:07 PM, John Dillon
<john.dillon892@googlemail•com> wrote:
> After all Peter, just like you have implemented alternate block header
> distribution over twitter, in the future we should have many different means of
> peer discovery. Right now we have DNS seeds, a fixed list, and IRC discovery
> that does not work because the servers it was pointed too no longer exist. Not
> a good place to be.
Let's not confuse bootstrapping with overall peer discovery.
Peer exchange between P2P nodes is the primary and best method of
obtaining free peers.
Obviously you need to bootstrap into that, though. DNS seed and fixed
list are those bootstrap methods (IRC code was deleted), but are only
used to limp along until you can contact a real P2P node, at which
point peer discovery truly begins.
--
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti•com
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-03 15:11 ` Peter Todd
@ 2013-05-04 18:07 ` John Dillon
2013-05-04 18:55 ` Jeff Garzik
0 siblings, 1 reply; 26+ messages in thread
From: John Dillon @ 2013-05-04 18:07 UTC (permalink / raw)
To: Peter Todd; +Cc: Bitcoin Dev
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
I think you too should ask yourself why you are putting so much effort into
optimizing a centralized service, the DNS seeds, rather than putting effort
into optimizing the P2P peer discovery instead. DNS seeds are a necessary evil,
one that shouldn't be promoted with additional features beyond simply obtaining
your initial set of peers.
After all Peter, just like you have implemented alternate block header
distribution over twitter, in the future we should have many different means of
peer discovery. Right now we have DNS seeds, a fixed list, and IRC discovery
that does not work because the servers it was pointed too no longer exist. Not
a good place to be.
Some random ideas:
search engines - search for "bitcoin seed address" or something and try IP's
found (twitter is similar)
ipv4 scanning - not exactly friendly, but the density of bitcoin nodes is
probably getting to the point where a brute force search is feasible
anycast peers - would work best with UDP probably, who has the resources to set
this up?
It is probably not worth the effort implementing the above immediately, but it
is worth the effort to ensure that we don't make the DNS seed system so complex
and sophisticated that we depend on it.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBCAAGBQJRhU44AAoJEEWCsU4mNhiPtssH/1yb/FZRRaZpr3CwkoaOhbhu
pxfRNWgOEOL/mlWKTVgp2812qEnY9DySpJ5DJMjx7/GhSvOtnteza5ts4+pbuWhd
l6E1R9zAYxX+VOiBxcBtoZNEXDcS+CjMumuBH5S1v+L5jEntOWS9G8DKasjD2WAQ
DzX8YbOuzIOqasEbr5Hpr9Vfl7ZtW/+q/sPhQ1q3a7n7MaaIZrZicisJw3z7T7+0
T0yK8vUdYfstTjs0zLzfI5PW9+TG5T0kvj0kXSCjnK723Mfl7SXp6UZx6yebBi6q
tcTVOPo4hfBWk8XryZxaSNCkDYY6kryy5cb2V+BojVfqLWVKgR3pdZqXqnEKNLo=
=0XFF
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-03 15:02 ` Mike Hearn
@ 2013-05-03 15:11 ` Peter Todd
2013-05-04 18:07 ` John Dillon
0 siblings, 1 reply; 26+ messages in thread
From: Peter Todd @ 2013-05-03 15:11 UTC (permalink / raw)
To: Mike Hearn; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]
On Fri, May 03, 2013 at 05:02:26PM +0200, Mike Hearn wrote:
> > If you're going to take a step like that, the <current-chain-height>
> > should be rounded off, perhaps to some number of bits, or you'll allow
> > DNS caching to be defeated.
> >
>
> Don't the seeds already set small times? I'm not sure we want these
> responses to be cacheable, otherwise there's a risk of a wall of traffic
> suddenly showing up at one set of nodes if a large ISP caches a response.
> (yes yes, I know, SPV node should be remembering addr broadcasts and such).
Hmm, on second thought you're probably right for the standard case where
it's really P2P. On the other hand it kinda limits us in the future if
seeds have high-bandwidth nodes they can just point clients too, but
maybe just assuming the DNS seed might need high bandwidth as well is
acceptable.
I dunno, given how badly behaved a lot of ISP dns servers are re:
caching, maybe we're better off keeping it simple.
--
'peter'[:-1]@petertodd.org
000000000000013bfdf35da40a40c35ccd75e09652ae541d94d26130a695f757
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-03 14:18 ` Peter Todd
@ 2013-05-03 15:02 ` Mike Hearn
2013-05-03 15:11 ` Peter Todd
0 siblings, 1 reply; 26+ messages in thread
From: Mike Hearn @ 2013-05-03 15:02 UTC (permalink / raw)
To: Peter Todd; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 468 bytes --]
> If you're going to take a step like that, the <current-chain-height>
> should be rounded off, perhaps to some number of bits, or you'll allow
> DNS caching to be defeated.
>
Don't the seeds already set small times? I'm not sure we want these
responses to be cacheable, otherwise there's a risk of a wall of traffic
suddenly showing up at one set of nodes if a large ISP caches a response.
(yes yes, I know, SPV node should be remembering addr broadcasts and such).
[-- Attachment #2: Type: text/html, Size: 758 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-03 14:06 ` Mike Hearn
@ 2013-05-03 14:18 ` Peter Todd
2013-05-03 15:02 ` Mike Hearn
0 siblings, 1 reply; 26+ messages in thread
From: Peter Todd @ 2013-05-03 14:18 UTC (permalink / raw)
To: Mike Hearn; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]
On Fri, May 03, 2013 at 04:06:29PM +0200, Mike Hearn wrote:
> That's true, but we can extend the DNS seeding protocol a little bit - you
> could query <current-chain-height>.dnsseed.whatever.com and the DNS server
> then only returns nodes it knows matches your requirement.
If you're going to take a step like that, the <current-chain-height>
should be rounded off, perhaps to some number of bits, or you'll allow
DNS caching to be defeated.
Make clients check for the largest "rounded off" value first, and then
drill down if required. Some complexity involved...
> This might complicate existing seeds a bit, and it's a bit of a hack, but
> protocol-wise it's still possible. Of course if you want to add more
> dimensions it gets uglier fast.
Maybe I should make my blockheaders-over-dns thing production worthy
first so we can see how many ISP's come at us with pitchforks? :P
--
'peter'[:-1]@petertodd.org
00000000000000142de0244ee8fac516e7c0a29da1eafc0d43f2da8b6388b387
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-05-03 12:30 ` Pieter Wuille
@ 2013-05-03 14:06 ` Mike Hearn
2013-05-03 14:18 ` Peter Todd
0 siblings, 1 reply; 26+ messages in thread
From: Mike Hearn @ 2013-05-03 14:06 UTC (permalink / raw)
To: Pieter Wuille; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 900 bytes --]
> Yes, I like that better than broadcasting the exact height starting at
> which you serve (though I would put that information immediately in the
> version announcement). I don't think we can rely on the addr broadcasting
> mechanism for fast information exchange anyway. One more problem with this:
> DNS seeds cannot convey this information (neither do they currently convey
> service bits, but at least those can be indexed separately, and served
> explicitly through asking for a specific subdomain or so).
>
That's true, but we can extend the DNS seeding protocol a little bit - you
could query <current-chain-height>.dnsseed.whatever.com and the DNS server
then only returns nodes it knows matches your requirement.
This might complicate existing seeds a bit, and it's a bit of a hack, but
protocol-wise it's still possible. Of course if you want to add more
dimensions it gets uglier fast.
[-- Attachment #2: Type: text/html, Size: 1345 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-28 16:57 ` Mike Hearn
@ 2013-05-03 12:30 ` Pieter Wuille
2013-05-03 14:06 ` Mike Hearn
0 siblings, 1 reply; 26+ messages in thread
From: Pieter Wuille @ 2013-05-03 12:30 UTC (permalink / raw)
To: Mike Hearn; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 3530 bytes --]
(generic comment on the discussion that spawned off: ideas about how to
allow additional protocols for block exchange are certainly interesting,
and in the long term we should certainly consider that. For now I'd like to
keep this about the more immediate way forward with making the P2P protocol
not break in the presence of pruning nodes)
On Sun, Apr 28, 2013 at 6:57 PM, Mike Hearn <mike@plan99•net> wrote:
> That's true. It can be perhaps be represented as "I keep the last N
> blocks" and then most likely for any given node the policy doesn't change
> all that fast, so if you know the best chain height you can calculate which
> nodes have what.
>
Yes, I like that better than broadcasting the exact height starting at
which you serve (though I would put that information immediately in the
version announcement). I don't think we can rely on the addr broadcasting
mechanism for fast information exchange anyway. One more problem with this:
DNS seeds cannot convey this information (neither do they currently convey
service bits, but at least those can be indexed separately, and served
explicitly through asking for a specific subdomain or so).
So to summarize:
* Add a field to addr messages (after protocol number increase) that
maintains number of top blocks served)?
* Add a field to version message to announce the actual first block served?
* Add service bits to separately enable "relaying/verifying node" and
"serves (part of) the historic chain"? My original reason for suggesting
this was different, I think better compatibility with DNS seeds may be a
good reason for this. You could ask the seed first for a subset that at
least serves some part of the historic chain, until you hit a node that has
enough, and once caught up, ask for nodes that relay.
Disconnecting in case something is requested that isn't served seems like
>> an acceptable behaviour, yes. A specific message indicating data is pruned
>> may be more flexible, but more complex to handle too.
>>
>
> Well, old nodes would ignore it and new nodes wouldn't need it?
>
I'm sure there will be cases where a new node connects based on outdated
information. I'm just stating that I agree with the generic policy of "if a
node requests something it should have known the peer doesn't serve, it is
fair to be disconnected."
> The reason for splitting them is that I think over time these may be
>> handled by different implementations. You could have stupid
>> storage/bandwidth nodes that just keep the blockchain around, and others
>> that validate it. Even if that doesn't happen implementation-wise, I think
>> these are sufficiently independent functions to start thinking about them
>> as such.
>>
>
> Maybe so, with a "last N blocks" in addr messages though such nodes could
> just set their advertised history to zero and not have to deal with serving
> blocks to nodes.
>
> If you have a node that serves the chain but doesn't validate it, how does
> it know what the best chain is? Just whatever the hardest is?
>
Maybe it validates, maybe it doesn't. What matters is that it doesn't
guarantee relaying fresh blocks and transactions. Maybe it does validate,
maybe it just stores any blocks, and uses a validating node to know what to
announce as best chain, or it uses an SPV mechanism to determine that. Or
it only validates and relays blocks, but not transactions. My point is that
"serving historic data" and "relaying fresh data" are separate
responsibilities, and there's no need to require them to be combined.
--
Pieter
[-- Attachment #2: Type: text/html, Size: 5880 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-28 15:51 Pieter Wuille
2013-04-28 16:29 ` Mike Hearn
@ 2013-05-01 13:46 ` Jeff Garzik
1 sibling, 0 replies; 26+ messages in thread
From: Jeff Garzik @ 2013-05-01 13:46 UTC (permalink / raw)
To: Pieter Wuille; +Cc: Bitcoin Dev
On Sun, Apr 28, 2013 at 11:51 AM, Pieter Wuille <pieter.wuille@gmail•com> wrote:
> Hello all,
>
> I think it is time to move forward with pruning nodes, i.e. nodes that fully
> validate and relay blocks and transactions, but which do not keep (all)
> historic blocks around, and thus cannot be queried for these.
>
> The biggest roadblock is making sure new and old nodes that start up are
> able to find nodes to synchronize from. To help them find peers, I would
> like to propose adding two extra service bits to the P2P protocol:
> * NODE_VALIDATE: relay and validate blocks and transactions, but is only
> guaranteed to answer getdata requests for (recently) relayed blocks and
> transactions, and mempool transactions.
> * NODE_BLOCKS_2016: can be queried for the last 2016 blocks, but without
> guarantee for relaying/validating new blocks and transactions.
> * NODE_NETWORK (which existed before) will imply NODE_VALIDATE and guarantee
> availability of all historic blocks.
In general, I support this, as anybody on IRC knows.
Though it does seem to open the question about snapshotting.
Personally, it seems important to enable running a fully validating
node, that may bootstrap from a UTXO snapshot + all blocks since that
snapshot.
NODE_BLOCKS_2016, in particular, is too short. For users, I've seen
plenty of use cases in the field where you start a network sync after
a 2-week period.
Set a regular interval for creating a UTXO snapshot, say 3 months
(6*2016 blocks), and serve all blocks after that snapshot. For older
nodes, they would contact an archive node or torrent for >3 month
blocks, and then download normally <= 3 month blocks (if the archive
node didn't serve up to present day).
Where are we on nailing down a stable, hash-able UTXO serialization?
--
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti•com
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-30 18:04 ` Jeff Garzik
@ 2013-04-30 20:06 ` Brenton Camac
0 siblings, 0 replies; 26+ messages in thread
From: Brenton Camac @ 2013-04-30 20:06 UTC (permalink / raw)
To: bitcoin-development
Sounds like this part of Bitcoin (block sharing) would definitely benefit from having a REST (HTTP) API.
REST-based web APIs are a common feature of most online services these days. Makes writing other client services so much easier. Plus you get the benefit of the HTTP ecosystem for free (HTTP caches, etc).
- Brenton Camac
On Apr 30, 2013, at 1:04 PM, Jeff Garzik <jgarzik@exmulti•com> wrote:
> On Tue, Apr 30, 2013 at 12:14 PM, Rebroad (sourceforge)
> <rebroad+sourceforge.net@gmail•com> wrote:
>> As part of a roadmap for block downloading, I think this may be a good time
>> to look into providing an HTTP/HTTPS protocol for block downloading - this
>> would also allow web proxies to cache blocks and thus make it more
>> accessible, as well as cater for resumeable downloads.
>
> Speaking generally, I've always been a supporter of finding new and
> creative ways to store and transmit blocks. The more diversity, the
> less likely bitcoin can be shut down worldwide.
>
> HTTP is fine, but you run into many issues with large files. You
> would need a very well defined HTTP-retrievable layout, with proper
> HTTP headers along the entire path, if you want web caches to function
> properly. You need HTTP byte range support, HTTP 1.1 keep-alives, and
> other features for resuming large, interrupted downloads.
>
> The format currently used by bitcoind would be just fine --
> blocks/blkNNNN.dat for raw data, size-limited well below 1GB. Just
> need to add a small metadata download, and serve the raw block files.
>
> --
> Jeff Garzik
> exMULTI, Inc.
> jgarzik@exmulti•com
>
> ------------------------------------------------------------------------------
> Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET
> Get 100% visibility into your production application - at no cost.
> Code-level diagnostics for performance bottlenecks with <2% overhead
> Download for free and get started troubleshooting in minutes.
> http://p.sf.net/sfu/appdyn_d2d_ap1
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-29 3:55 ` Peter Todd
@ 2013-04-29 6:10 ` Jay F
[not found] ` <CAFBxzACw=G7UgG853zQrM-Z1-B4VqSQR5YUJQ5n1=wnv7EyWsw@mail.gmail.com>
0 siblings, 1 reply; 26+ messages in thread
From: Jay F @ 2013-04-29 6:10 UTC (permalink / raw)
To: Peter Todd; +Cc: Bitcoin Dev
On 4/28/2013 8:55 PM, Peter Todd wrote:
> On Mon, Apr 29, 2013 at 03:48:18AM +0000, John Dillon wrote:
>> We can build this stuff incrementally I'll agree. It won't be the case that one
>> in a thousand nodes serve up the part of the chain you need overnight. So many
>> I am over engineering the solution with BitTorrent.
>
> I think that pretty much sums it up.
>
> With the block-range served in the anounce message you just need to find
> an annoucement with the right range, and at worst connect to a few more
> node to get what you need.
One of the technologies that can be borrowed from Bittorrent (besides
downloading from multiple peers at once) is analysis by clients of the
part distribution, which allows a client to download and share the
least-propagated parts first to maintain high availability of the whole
file, even when not one individual currently has downloaded the complete
file (the seed has left the swarm).
Unlike Bittorrent, a partial-blockchain swarm client needs to make
informed decisions about how much to download, such as rules like "until
it sees at least 20 complete blockchain-equivalents in the swarm",
"until it has 10% of the blockchain itself", "work backwards, all blocks
from the hash tree required to verify my payments" or other minimums
that might all be criteria.
Bittorrent only considers directly connected peers' piecemaps when
making decisions of what to download. Bitcoin, however, already has a
protocol to allow peer discovery beyond the connected nodes; this could
be extended to communicate what parts the peer is hosting. Careful
thought into attack vectors would need to be paid in design, so that
only a majority of outbound-connected peers's advertisement are able to
inform consensus about part or peer availability, messages able to
remove a peer or part availability from other's lists are confirmed
independently without such removal verification generating DDOS traffic
amplification, lying clients can be marked as discovered by the
majority, etc.
Such thought doesn't have to be paid if directly implementing
Bittorrent, but it has the burden of centralized trackers or expensive
DHT, and it also doesn't have any logic informing it besides "don't quit
until I get the whole file".
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-29 3:48 ` John Dillon
@ 2013-04-29 3:55 ` Peter Todd
2013-04-29 6:10 ` Jay F
0 siblings, 1 reply; 26+ messages in thread
From: Peter Todd @ 2013-04-29 3:55 UTC (permalink / raw)
To: John Dillon; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]
On Mon, Apr 29, 2013 at 03:48:18AM +0000, John Dillon wrote:
> We can build this stuff incrementally I'll agree. It won't be the case that one
> in a thousand nodes serve up the part of the chain you need overnight. So many
> I am over engineering the solution with BitTorrent.
I think that pretty much sums it up.
With the block-range served in the anounce message you just need to find
an annoucement with the right range, and at worst connect to a few more
node to get what you need. It will be a long time before the bandwidth
used for finding a node with the part of the chain that you need is a
significant fraction of the load required for downloading the data
itself.
Remember that BitTorrent's DHT is a system giving you access to tens of
petabytes worth of data. The Bitcoin blockchain on the other hand simply
can't grow more than 57GiB per year. It's a cute idea though.
Also, while we're talking about the initial download:
http://blockchainbymail.com
Lots of options out there.
--
'peter'[:-1]@petertodd.org
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-29 3:36 ` Gregory Maxwell
2013-04-29 3:42 ` Robert Backhaus
@ 2013-04-29 3:48 ` John Dillon
2013-04-29 3:55 ` Peter Todd
1 sibling, 1 reply; 26+ messages in thread
From: John Dillon @ 2013-04-29 3:48 UTC (permalink / raw)
To: Gregory Maxwell; +Cc: Bitcoin Dev
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Mon, Apr 29, 2013 at 3:36 AM, Gregory Maxwell <gmaxwell@gmail•com> wrote:
> On Sun, Apr 28, 2013 at 7:57 PM, John Dillon
> <john.dillon892@googlemail•com> wrote:
>> Have we considered just leaving that problem to a different protocol such as
>> BitTorrent? Offering up a few GB of storage capacity is a nice idea but it
>> means we would soon have to add structure to the network to allow nodes to find
>> each other to actually get that data. BitTorrent already has that issue thought
>> through carefully with it's DHT support.
>
> I think this is not a great idea on a couple levels—
>
> Least importantly, our own experience with tracker-less torrents on
> the bootstrap files that they don't work very well in practice— and
> thats without someone trying to DOS attack it.
Unfortunate. What makes them not work out? DHT torrents seem pretty popular.
> More importantly, I think it's very important that the process of
> offering up more storage not take any more steps. The software could
> have user overridable defaults based on free disk space to make
> contributing painless. This isn't possible if it takes extra software,
> requires opening additional ports.. etc. Also means that someone
> would have to be constantly creating new torrents, there would be
> issues with people only seeding the old ones, etc.
Now don't get me wrong, I'm not proposing we do this if it requires additional
steps or other software. I only mean if it is possible in an easy way to
integrate the BitTorrent technology into Bitcoin in an automatic fashion. Yes
part of that may have to be finding a way to re-use the existing port for
instance.
> We already have to worry about nodes finding each other just for basic
> operation. The only addition this requires is being able to advertise
> what parts of the chain they have.
Sure I guess my concern is more how do you find the specific part of the chian
you need without some structure to the network? Although I guess it may be
enough to just add that structure or depend on just walking the nodes
advertising themselves until you find what you want.
We can build this stuff incrementally I'll agree. It won't be the case that one
in a thousand nodes serve up the part of the chain you need overnight. So many
I am over engineering the solution with BitTorrent.
> Using Bitcoin to bootstrap the Bittorrent DHT would probably make it
> more reliable, but then again it might cause commercial services that
> are in the business of poisoning the bittorrent DHT to target the
> Bitcoin network.
Good point. Sadly one that may apply to the Tor network too in the future.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBCAAGBQJRfe1LAAoJEEWCsU4mNhiPuDgIAM1zz+ohlHgz37RgToQhInRc
1tv4Fnb6uGWyb4+U6UpK24LlXMFvOJsLm2czgbBc1Iz4z4wvb1m5IGw0ubJuV4mT
GPUJhM4sNqfeKZlSWRw4Gia6Vk1jTkue+uVYvZn2vBS4SS6vYhYCC3zXIITyb2mp
7CVjcM84bTHKxIaMW1rIgmVJmfslsFdeNOp/cDVvkNl9+WvzWPeJ32BkT522p+pT
AcPVFMsEJirYrXYi8HwdtGSeiG+mv0IemTAObJNPRrpw3x04ja6qecqzM51AkQ4t
hPems5ShXM9FyDKFQNmtoC6ULpbd3CBBjsiQj0pp55epy6UC0eiUIXP8L9v0giM=
=AOj8
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-29 3:36 ` Gregory Maxwell
@ 2013-04-29 3:42 ` Robert Backhaus
2013-04-29 3:48 ` John Dillon
1 sibling, 0 replies; 26+ messages in thread
From: Robert Backhaus @ 2013-04-29 3:42 UTC (permalink / raw)
To: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 3283 bytes --]
While I like the idea of a client using a DHT blockchain or UTXO list, I
don't think that the reference client is the place for it. But it would
make for a very interesting experimental project!
On 29 April 2013 13:36, Gregory Maxwell <gmaxwell@gmail•com> wrote:
> On Sun, Apr 28, 2013 at 7:57 PM, John Dillon
> <john.dillon892@googlemail•com> wrote:
> > Have we considered just leaving that problem to a different protocol
> such as
> > BitTorrent? Offering up a few GB of storage capacity is a nice idea but
> it
> > means we would soon have to add structure to the network to allow nodes
> to find
> > each other to actually get that data. BitTorrent already has that issue
> thought
> > through carefully with it's DHT support.
>
> I think this is not a great idea on a couple levels—
>
> Least importantly, our own experience with tracker-less torrents on
> the bootstrap files that they don't work very well in practice— and
> thats without someone trying to DOS attack it.
>
> More importantly, I think it's very important that the process of
> offering up more storage not take any more steps. The software could
> have user overridable defaults based on free disk space to make
> contributing painless. This isn't possible if it takes extra software,
> requires opening additional ports.. etc. Also means that someone
> would have to be constantly creating new torrents, there would be
> issues with people only seeding the old ones, etc.
>
> It's also the case that bittorrent is blocked on many networks and is
> confused with illicit copying. We would have the same problems with
> that that we had with IRC being confused with botnets.
>
> We already have to worry about nodes finding each other just for basic
> operation. The only addition this requires is being able to advertise
> what parts of the chain they have.
>
> > What are the logistics of either integrating a DHT capable BitTorrent
> client,
> > or just calling out to some library? We could still use the Bitcoin
> network to
> > bootstrap the BitTorrent DHT.
>
> Using Bitcoin to bootstrap the Bittorrent DHT would probably make it
> more reliable, but then again it might cause commercial services that
> are in the business of poisoning the bittorrent DHT to target the
> Bitcoin network.
>
> Integration also brings up the question of network exposed attack surface.
>
> Seems like it would be more work than just adding the ability to add
> ranges to address messages. I think we already want to revise the
> address message format in order to have signed flags and to support
> I2P peers.
>
>
> ------------------------------------------------------------------------------
> Try New Relic Now & We'll Send You this Cool Shirt
> New Relic is the only SaaS-based application performance monitoring service
> that delivers powerful full stack analytics. Optimize and monitor your
> browser, app, & servers with just a few lines of code. Try New Relic
> and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 4108 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-29 2:57 ` John Dillon
@ 2013-04-29 3:36 ` Gregory Maxwell
2013-04-29 3:42 ` Robert Backhaus
2013-04-29 3:48 ` John Dillon
0 siblings, 2 replies; 26+ messages in thread
From: Gregory Maxwell @ 2013-04-29 3:36 UTC (permalink / raw)
To: John Dillon; +Cc: Bitcoin Dev
On Sun, Apr 28, 2013 at 7:57 PM, John Dillon
<john.dillon892@googlemail•com> wrote:
> Have we considered just leaving that problem to a different protocol such as
> BitTorrent? Offering up a few GB of storage capacity is a nice idea but it
> means we would soon have to add structure to the network to allow nodes to find
> each other to actually get that data. BitTorrent already has that issue thought
> through carefully with it's DHT support.
I think this is not a great idea on a couple levels—
Least importantly, our own experience with tracker-less torrents on
the bootstrap files that they don't work very well in practice— and
thats without someone trying to DOS attack it.
More importantly, I think it's very important that the process of
offering up more storage not take any more steps. The software could
have user overridable defaults based on free disk space to make
contributing painless. This isn't possible if it takes extra software,
requires opening additional ports.. etc. Also means that someone
would have to be constantly creating new torrents, there would be
issues with people only seeding the old ones, etc.
It's also the case that bittorrent is blocked on many networks and is
confused with illicit copying. We would have the same problems with
that that we had with IRC being confused with botnets.
We already have to worry about nodes finding each other just for basic
operation. The only addition this requires is being able to advertise
what parts of the chain they have.
> What are the logistics of either integrating a DHT capable BitTorrent client,
> or just calling out to some library? We could still use the Bitcoin network to
> bootstrap the BitTorrent DHT.
Using Bitcoin to bootstrap the Bittorrent DHT would probably make it
more reliable, but then again it might cause commercial services that
are in the business of poisoning the bittorrent DHT to target the
Bitcoin network.
Integration also brings up the question of network exposed attack surface.
Seems like it would be more work than just adding the ability to add
ranges to address messages. I think we already want to revise the
address message format in order to have signed flags and to support
I2P peers.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-28 19:50 ` Gregory Maxwell
@ 2013-04-29 2:57 ` John Dillon
2013-04-29 3:36 ` Gregory Maxwell
0 siblings, 1 reply; 26+ messages in thread
From: John Dillon @ 2013-04-29 2:57 UTC (permalink / raw)
To: Gregory Maxwell; +Cc: Bitcoin Dev
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
> But I also agree that its important that <everything> be splittable into ranges
> because otherwise when having to choose between serving historic data
> and— say— 40 GB storage, a great many are going to choose not to serve
> historic data... and so nodes may be willing to contribute 4-39 GB storage
> to the network there will be no good way for them to do so and we may end
> up with too few copies of the historic data available.
Have we considered just leaving that problem to a different protocol such as
BitTorrent? Offering up a few GB of storage capacity is a nice idea but it
means we would soon have to add structure to the network to allow nodes to find
each other to actually get that data. BitTorrent already has that issue thought
through carefully with it's DHT support.
What are the logistics of either integrating a DHT capable BitTorrent client,
or just calling out to some library? We could still use the Bitcoin network to
bootstrap the BitTorrent DHT.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBCAAGBQJRfeF/AAoJEEWCsU4mNhiP228H+gIiBhOf65ydmVXoq7d6toNt
FmWZaHUxAKtdADINqKHKjuCVGb+3RITwZIgQ0t2MO3OlG1FRFzZv841QBmdaW7JI
B6uF2hBxw6oy3GolzIbSUBX+7VyoNvFGT9c548wfLWC71O7A9/Wf3dUssN6VdWXG
zm2vTO8cnMOHNt0Iu4uRw5mvOOU6WV9f6k3BsnQEK8y8E3w1k8xZIrHMqCo99B5U
a0R2TOpIyK++8xz3Ls1johcFcfwkphESn8SIxMeyb/sgotxO23yqQNDqn8rDCD4S
PxVY/yzpftinjR55bvvjRGDVkUY43ixU8t7lFOgI1vwmfRw4jBqk7WWYJK7jC6c=
=0VmS
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-28 16:29 ` Mike Hearn
2013-04-28 16:44 ` Pieter Wuille
@ 2013-04-28 19:50 ` Gregory Maxwell
2013-04-29 2:57 ` John Dillon
1 sibling, 1 reply; 26+ messages in thread
From: Gregory Maxwell @ 2013-04-28 19:50 UTC (permalink / raw)
To: Mike Hearn; +Cc: Bitcoin Dev
On Sun, Apr 28, 2013 at 9:29 AM, Mike Hearn <mike@plan99•net> wrote:
> I'd imagined that nodes would be able to pick their own ranges to keep
> rather than have fixed chosen intervals. "Everything or two weeks" is rather
X most recent is special for two reasons: It meshes well with actual demand,
and the data is required for reorganization.
So whatever we do for historic data, N most recent should be treated
specially.
But I also agree that its important that <everything> be splittable into ranges
because otherwise when having to choose between serving historic data
and— say— 40 GB storage, a great many are going to choose not to serve
historic data... and so nodes may be willing to contribute 4-39 GB storage
to the network there will be no good way for them to do so and we may end
up with too few copies of the historic data available.
As can be seen in the graph, once you get past the most recent 4000
blocks the probability is fairly uniform... so "N most recent" is not a
good way to divide load for the older blocks. But simple ranges— perhaps
quantized to groups of 100 or 1000 blocks or something— would work fine.
This doesn't have to come in the first cut, however— and it needs new
addr messages in any case.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-28 16:44 ` Pieter Wuille
@ 2013-04-28 16:57 ` Mike Hearn
2013-05-03 12:30 ` Pieter Wuille
0 siblings, 1 reply; 26+ messages in thread
From: Mike Hearn @ 2013-04-28 16:57 UTC (permalink / raw)
To: Pieter Wuille; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1195 bytes --]
That's true. It can be perhaps be represented as "I keep the last N blocks"
and then most likely for any given node the policy doesn't change all that
fast, so if you know the best chain height you can calculate which nodes
have what.
> Disconnecting in case something is requested that isn't served seems like
> an acceptable behaviour, yes. A specific message indicating data is pruned
> may be more flexible, but more complex to handle too.
>
Well, old nodes would ignore it and new nodes wouldn't need it?
> The reason for splitting them is that I think over time these may be
> handled by different implementations. You could have stupid
> storage/bandwidth nodes that just keep the blockchain around, and others
> that validate it. Even if that doesn't happen implementation-wise, I think
> these are sufficiently independent functions to start thinking about them
> as such.
>
Maybe so, with a "last N blocks" in addr messages though such nodes could
just set their advertised history to zero and not have to deal with serving
blocks to nodes.
If you have a node that serves the chain but doesn't validate it, how does
it know what the best chain is? Just whatever the hardest is?
[-- Attachment #2: Type: text/html, Size: 1994 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-28 16:29 ` Mike Hearn
@ 2013-04-28 16:44 ` Pieter Wuille
2013-04-28 16:57 ` Mike Hearn
2013-04-28 19:50 ` Gregory Maxwell
1 sibling, 1 reply; 26+ messages in thread
From: Pieter Wuille @ 2013-04-28 16:44 UTC (permalink / raw)
To: Mike Hearn; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]
On Sun, Apr 28, 2013 at 6:29 PM, Mike Hearn <mike@plan99•net> wrote:
> I'd imagined that nodes would be able to pick their own ranges to keep
> rather than have fixed chosen intervals. "Everything or two weeks" is
> rather restrictive - presumably node operators are constrained by physical
> disk space, which means the quantity of blocks they would want to keep can
> vary with sizes of blocks, cost of storage, etc.
>
Sure, that's why eventually several levels may be useful.
Adding new fields to the addr message and relaying those fields to newer
> nodes means every node could advertise the height at which it pruned. I
> know it means a longer time before the data is available everywhere vs
> service bits, but it seems like most nodes won't be pruning right away
> anyway. There's plenty of time for upgrades.
>
That's a more flexible model, indeed. I'm not sure how important speed of
propagation will be though - it may be very slow, given that there are
100000s of IPs circulating, and only a few are relayed in one go between
nodes. Even then, I'd like to see the "relay/validation" responsibility
split off from the "serve historic data" one, and have separate service
bits for those.
> If an old node connected to a new node and getdata-d blocks that had been
> pruned, immediate disconnection should make the old node go find a
> different one. It means the combination of old node+not run for a long time
> might take a while before it can find a node that has what it wants, but
> that doesn't seem like a big deal.
>
Disconnecting in case something is requested that isn't served seems like
an acceptable behaviour, yes. A specific message indicating data is pruned
may be more flexible, but more complex to handle too.
What is the use case for NODE_VALIDATE? Nodes that throw away blocks almost
> immediately? Why would a node do that?
>
NODE_VALIDATE doesn't say anything about which blocks are available, it
just means it relays and validates (and thus is not an SPV node). It can be
combined with NODE_BLOCKS_2016 if those blocks are also served.
The reason for splitting them is that I think over time these may be
handled by different implementations. You could have stupid
storage/bandwidth nodes that just keep the blockchain around, and others
that validate it. Even if that doesn't happen implementation-wise, I think
these are sufficiently independent functions to start thinking about them
as such.
--
Pieter
[-- Attachment #2: Type: text/html, Size: 3538 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Bitcoin-development] Service bits for pruned nodes
2013-04-28 15:51 Pieter Wuille
@ 2013-04-28 16:29 ` Mike Hearn
2013-04-28 16:44 ` Pieter Wuille
2013-04-28 19:50 ` Gregory Maxwell
2013-05-01 13:46 ` Jeff Garzik
1 sibling, 2 replies; 26+ messages in thread
From: Mike Hearn @ 2013-04-28 16:29 UTC (permalink / raw)
To: Pieter Wuille; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 3484 bytes --]
I'd imagined that nodes would be able to pick their own ranges to keep
rather than have fixed chosen intervals. "Everything or two weeks" is
rather restrictive - presumably node operators are constrained by physical
disk space, which means the quantity of blocks they would want to keep can
vary with sizes of blocks, cost of storage, etc.
Adding new fields to the addr message and relaying those fields to newer
nodes means every node could advertise the height at which it pruned. I
know it means a longer time before the data is available everywhere vs
service bits, but it seems like most nodes won't be pruning right away
anyway. There's plenty of time for upgrades. If an old node connected to a
new node and getdata-d blocks that had been pruned, immediate disconnection
should make the old node go find a different one. It means the combination
of old node+not run for a long time might take a while before it can find a
node that has what it wants, but that doesn't seem like a big deal.
What is the use case for NODE_VALIDATE? Nodes that throw away blocks almost
immediately? Why would a node do that?
On Sun, Apr 28, 2013 at 5:51 PM, Pieter Wuille <pieter.wuille@gmail•com>wrote:
> Hello all,
>
> I think it is time to move forward with pruning nodes, i.e. nodes that
> fully validate and relay blocks and transactions, but which do not keep
> (all) historic blocks around, and thus cannot be queried for these.
>
> The biggest roadblock is making sure new and old nodes that start up are
> able to find nodes to synchronize from. To help them find peers, I would
> like to propose adding two extra service bits to the P2P protocol:
> * NODE_VALIDATE: relay and validate blocks and transactions, but is only
> guaranteed to answer getdata requests for (recently) relayed blocks and
> transactions, and mempool transactions.
> * NODE_BLOCKS_2016: can be queried for the last 2016 blocks, but without
> guarantee for relaying/validating new blocks and transactions.
> * NODE_NETWORK (which existed before) will imply NODE_VALIDATE and
> guarantee availability of all historic blocks.
>
> The idea is to separate the different responsibilities of network nodes
> into separate bits, so they can - at some point - be
> implemented independently. Perhaps we want more than just one degree (2016
> blocks), maybe also 144 or 210000, but those can be added later if
> necessary. I monitored the frequency of block depths requested from my
> public node, and got this frequency distribution:
> http://bitcoin.sipa.be/depth-small.png so it seems 2016 nicely matches
> the set of frequently-requested blocks (indicating that few nodes are
> offline for more than 2 weeks consecutively.
>
> I'll write a BIP to formalize this, but wanted to get an idea of how much
> support there is for a change like this.
>
> Cheers,
>
> --
> Pieter
>
>
>
>
>
> ------------------------------------------------------------------------------
> Try New Relic Now & We'll Send You this Cool Shirt
> New Relic is the only SaaS-based application performance monitoring service
> that delivers powerful full stack analytics. Optimize and monitor your
> browser, app, & servers with just a few lines of code. Try New Relic
> and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
[-- Attachment #2: Type: text/html, Size: 4494 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* [Bitcoin-development] Service bits for pruned nodes
@ 2013-04-28 15:51 Pieter Wuille
2013-04-28 16:29 ` Mike Hearn
2013-05-01 13:46 ` Jeff Garzik
0 siblings, 2 replies; 26+ messages in thread
From: Pieter Wuille @ 2013-04-28 15:51 UTC (permalink / raw)
To: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]
Hello all,
I think it is time to move forward with pruning nodes, i.e. nodes that
fully validate and relay blocks and transactions, but which do not keep
(all) historic blocks around, and thus cannot be queried for these.
The biggest roadblock is making sure new and old nodes that start up are
able to find nodes to synchronize from. To help them find peers, I would
like to propose adding two extra service bits to the P2P protocol:
* NODE_VALIDATE: relay and validate blocks and transactions, but is only
guaranteed to answer getdata requests for (recently) relayed blocks and
transactions, and mempool transactions.
* NODE_BLOCKS_2016: can be queried for the last 2016 blocks, but without
guarantee for relaying/validating new blocks and transactions.
* NODE_NETWORK (which existed before) will imply NODE_VALIDATE and
guarantee availability of all historic blocks.
The idea is to separate the different responsibilities of network nodes
into separate bits, so they can - at some point - be
implemented independently. Perhaps we want more than just one degree (2016
blocks), maybe also 144 or 210000, but those can be added later if
necessary. I monitored the frequency of block depths requested from my
public node, and got this frequency distribution:
http://bitcoin.sipa.be/depth-small.png so it seems 2016 nicely matches the
set of frequently-requested blocks (indicating that few nodes are offline
for more than 2 weeks consecutively.
I'll write a BIP to formalize this, but wanted to get an idea of how much
support there is for a change like this.
Cheers,
--
Pieter
[-- Attachment #2: Type: text/html, Size: 1947 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2013-05-16 16:23 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-05-16 11:26 [Bitcoin-development] Service bits for pruned nodes Ricardo Filipe
2013-05-16 15:47 ` Jeff Garzik
2013-05-16 16:23 ` Ricardo Filipe
-- strict thread matches above, loose matches on Subject: below --
2013-04-28 15:51 Pieter Wuille
2013-04-28 16:29 ` Mike Hearn
2013-04-28 16:44 ` Pieter Wuille
2013-04-28 16:57 ` Mike Hearn
2013-05-03 12:30 ` Pieter Wuille
2013-05-03 14:06 ` Mike Hearn
2013-05-03 14:18 ` Peter Todd
2013-05-03 15:02 ` Mike Hearn
2013-05-03 15:11 ` Peter Todd
2013-05-04 18:07 ` John Dillon
2013-05-04 18:55 ` Jeff Garzik
2013-05-05 13:12 ` John Dillon
2013-05-06 8:19 ` Mike Hearn
2013-05-06 13:13 ` Pieter Wuille
2013-04-28 19:50 ` Gregory Maxwell
2013-04-29 2:57 ` John Dillon
2013-04-29 3:36 ` Gregory Maxwell
2013-04-29 3:42 ` Robert Backhaus
2013-04-29 3:48 ` John Dillon
2013-04-29 3:55 ` Peter Todd
2013-04-29 6:10 ` Jay F
[not found] ` <CAFBxzACw=G7UgG853zQrM-Z1-B4VqSQR5YUJQ5n1=wnv7EyWsw@mail.gmail.com>
2013-04-30 16:14 ` [Bitcoin-development] Fwd: " Rebroad (sourceforge)
2013-04-30 18:04 ` Jeff Garzik
2013-04-30 20:06 ` [Bitcoin-development] " Brenton Camac
2013-05-01 13:46 ` Jeff Garzik
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox