* [bitcoin-dev] Idea: Efficient bitcoin block propagation @ 2015-08-05 19:07 Arnoud Kouwenhoven - Pukaki Corp 2015-08-05 19:27 ` Matt Corallo 2015-08-07 7:14 ` jl2012 0 siblings, 2 replies; 14+ messages in thread From: Arnoud Kouwenhoven - Pukaki Corp @ 2015-08-05 19:07 UTC (permalink / raw) To: bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 2151 bytes --] Hello all. We’d like to share an idea we have to dramatically increase the bitcoin block propagation speed after a new block has been mined for the first time. Efficient bitcoin block propagation A proposed solution to provide near-instantaneous block propagation on the bitcoin network, even with slow network connections or large block sizes. Increasing mining efficiency for everyone while decreasing transaction confirmation times and strengthening the distributed nature of bitcoin. Short summary: we propose to introduce bitcoin-backed guarantees (“Guarantee Messages”) between miners. This would allow miners to mine on blocks that are not yet fully transmitted. This reduces the effect of slow internet connections, leveling the playing field between the 1st world fiberoptic datacenter miners and the rest of the world. We also believe it strengthens the bitcoin network by using existing processing power that is currently wasted into further securing the blockchain, and it reduces the likelihood of transactions becoming confirmed, then unconfirmed and then -hopefully- confirmed again (due to different miners finding competing blocks with different transactions at approx the same time). It is possible to implement our idea as a fork of bitcoind, or as layer between the standard bitcoind and the mining equipment. In the future it could be incorporated in the bitcoin core if and when that becomes a priority, but that step would not make sense until it becomes a priority. There are a lot of nuances in this idea, and the first reaction is quite probably that this is a crazy idea. We have attempted to address the most important nuances in our proposal, which is currently at v.0.2. We cannot guarantee that there are no ‘hidden devils in the details’ and we invite you to be critical in a friendly and constructive manner. We will do our best to answer all questions that arise. The ‘official’ proposal is at: PDF: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.pdf HTML: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.html -- Arnoud Kouwenhoven [-- Attachment #2: Type: text/html, Size: 2367 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-05 19:07 [bitcoin-dev] Idea: Efficient bitcoin block propagation Arnoud Kouwenhoven - Pukaki Corp @ 2015-08-05 19:27 ` Matt Corallo 2015-08-05 19:53 ` Arnoud Kouwenhoven - Pukaki Corp 2015-08-07 7:14 ` jl2012 1 sibling, 1 reply; 14+ messages in thread From: Matt Corallo @ 2015-08-05 19:27 UTC (permalink / raw) To: Arnoud Kouwenhoven - Pukaki Corp, Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev, bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 3423 bytes --] See-also: Bitcoinrelaynetwork.org. It's already in use my the majority of large miners, is publicly available to anyone, and the protocol is rather simple and the client could be tweaked easily to keep exactly it's block ready to quickly relay to the nearest server (ie only have to relay the header, the coinbase transaction, and only small other data... Experience shows this is really easy to fit into one packet on the wire). It's not nearly as complicated as your suggestion, but may still marginally favor well-connected miners, but hopefully not much (when you're taking about single packets, it should all be latency, and the servers are well distributed). If you feel so inclined, there are some todos to make it really meet is efficiency limits filled on github.com/TheBlueMatt/RelayNode, feel free to rewrite the protocol if you really want :). Matt On August 5, 2015 9:07:44 PM GMT+02:00, Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: >Hello all. > >We’d like to share an idea we have to dramatically increase the bitcoin >block propagation speed after a new block has been mined for the first >time. > >Efficient bitcoin block propagation >A proposed solution to provide near-instantaneous block propagation on >the >bitcoin network, even with slow network connections or large block >sizes. >Increasing mining efficiency for everyone while decreasing transaction >confirmation times and strengthening the distributed nature of bitcoin. > >Short summary: we propose to introduce bitcoin-backed guarantees >(“Guarantee Messages”) between miners. This would allow miners to mine >on >blocks that are not yet fully transmitted. This reduces the effect of >slow >internet connections, leveling the playing field between the 1st world >fiberoptic datacenter miners and the rest of the world. We also believe >it >strengthens the bitcoin network by using existing processing power that >is >currently wasted into further securing the blockchain, and it reduces >the >likelihood of transactions becoming confirmed, then unconfirmed and >then >-hopefully- confirmed again (due to different miners finding competing >blocks with different transactions at approx the same time). > >It is possible to implement our idea as a fork of bitcoind, or as layer >between the standard bitcoind and the mining equipment. In the future >it >could be incorporated in the bitcoin core if and when that becomes a >priority, but that step would not make sense until it becomes a >priority. > >There are a lot of nuances in this idea, and the first reaction is >quite >probably that this is a crazy idea. We have attempted to address the >most >important nuances in our proposal, which is currently at v.0.2. > >We cannot guarantee that there are no ‘hidden devils in the details’ >and we >invite you to be critical in a friendly and constructive manner. We >will do >our best to answer all questions that arise. > >The ‘official’ proposal is at: >PDF: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.pdf >HTML: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.html > >-- Arnoud Kouwenhoven > > >------------------------------------------------------------------------ > >_______________________________________________ >bitcoin-dev mailing list >bitcoin-dev@lists•linuxfoundation.org >https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [-- Attachment #2: Type: text/html, Size: 4096 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-05 19:27 ` Matt Corallo @ 2015-08-05 19:53 ` Arnoud Kouwenhoven - Pukaki Corp 2015-08-05 20:16 ` Gregory Maxwell 0 siblings, 1 reply; 14+ messages in thread From: Arnoud Kouwenhoven - Pukaki Corp @ 2015-08-05 19:53 UTC (permalink / raw) To: Matt Corallo; +Cc: Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 4752 bytes --] Thanks for the reply. My understanding is that the bitcoin relay network is a backbone of connected high speed servers to increase the rate at which transactions and new blocks propagate - and remove a number of delays in processing. But it would still require the miners to download the entire block before building on top of it with any degree of confidence. With a tweak to only send the required information for other miners to build on top of that block, this is a step towards what we propose, yet would require trust that the header information sent is accurate. The bitcoin relay network website states that blocks are not fully verified and should be checked by the miners before building on top of them. What we propose is more complex (granted!), yet that complexity serves a purpose. We reduce (and hopefully eliminate) the adverse incentive to entice miners to build on inaccurate data. This is achieved by making the financial losses of fake messages outweigh the financial gains of such attack vectors. It could also help in the block size debate if this proposed solution would eliminate the disadvantages of large blocks. On Wed, Aug 5, 2015 at 1:27 PM, Matt Corallo <lf-lists@mattcorallo•com> wrote: > See-also: Bitcoinrelaynetwork.org. It's already in use my the majority of > large miners, is publicly available to anyone, and the protocol is rather > simple and the client could be tweaked easily to keep exactly it's block > ready to quickly relay to the nearest server (ie only have to relay the > header, the coinbase transaction, and only small other data... Experience > shows this is really easy to fit into one packet on the wire). It's not > nearly as complicated as your suggestion, but may still marginally favor > well-connected miners, but hopefully not much (when you're taking about > single packets, it should all be latency, and the servers are well > distributed). If you feel so inclined, there are some todos to make it > really meet is efficiency limits filled on > github.com/TheBlueMatt/RelayNode, feel free to rewrite the protocol if > you really want :). > > Matt > > On August 5, 2015 9:07:44 PM GMT+02:00, Arnoud Kouwenhoven - Pukaki Corp > via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > >> Hello all. >> >> We’d like to share an idea we have to dramatically increase the bitcoin >> block propagation speed after a new block has been mined for the first time. >> >> Efficient bitcoin block propagation >> A proposed solution to provide near-instantaneous block propagation on >> the bitcoin network, even with slow network connections or large block >> sizes. Increasing mining efficiency for everyone while decreasing >> transaction confirmation times and strengthening the distributed nature of >> bitcoin. >> >> Short summary: we propose to introduce bitcoin-backed guarantees >> (“Guarantee Messages”) between miners. This would allow miners to mine on >> blocks that are not yet fully transmitted. This reduces the effect of slow >> internet connections, leveling the playing field between the 1st world >> fiberoptic datacenter miners and the rest of the world. We also believe it >> strengthens the bitcoin network by using existing processing power that is >> currently wasted into further securing the blockchain, and it reduces the >> likelihood of transactions becoming confirmed, then unconfirmed and then >> -hopefully- confirmed again (due to different miners finding competing >> blocks with different transactions at approx the same time). >> >> It is possible to implement our idea as a fork of bitcoind, or as layer >> between the standard bitcoind and the mining equipment. In the future it >> could be incorporated in the bitcoin core if and when that becomes a >> priority, but that step would not make sense until it becomes a priority. >> >> There are a lot of nuances in this idea, and the first reaction is quite >> probably that this is a crazy idea. We have attempted to address the most >> important nuances in our proposal, which is currently at v.0.2. >> >> We cannot guarantee that there are no ‘hidden devils in the details’ and >> we invite you to be critical in a friendly and constructive manner. We will >> do our best to answer all questions that arise. >> >> The ‘official’ proposal is at: >> PDF: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.pdf >> HTML: http://pukaki.bz/efficient-bitcoin-block-propagation-v.0.2.html >> >> -- Arnoud Kouwenhoven >> >> ------------------------------ >> >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >> [-- Attachment #2: Type: text/html, Size: 5833 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-05 19:53 ` Arnoud Kouwenhoven - Pukaki Corp @ 2015-08-05 20:16 ` Gregory Maxwell 2015-08-05 21:19 ` Arnoud Kouwenhoven - Pukaki Corp 0 siblings, 1 reply; 14+ messages in thread From: Gregory Maxwell @ 2015-08-05 20:16 UTC (permalink / raw) To: Arnoud Kouwenhoven - Pukaki Corp Cc: Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev On Wed, Aug 5, 2015 at 7:53 PM, Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > Thanks for the reply. My understanding is that the bitcoin relay network is > a backbone of connected high speed servers to increase the rate at which > transactions and new blocks propagate - and remove a number of delays in > processing. But it would still require the miners to download the entire > block before building on top of it with any degree of confidence. Your understanding is outdated. The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block). E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes. No trust is required in this scheme because the entire block is communicated using only a couple packets. The current scheme is highly simplified and its efficiency could be increased greatly with small improvements, or if miners created blocks in an aware manner.... but with a maximum size blocks turning into 5kb with the current setup, there hardly appears to be a reason to do so right now. Ultimately there is no need for information communicated with a block at discovery time proportional to the size of the block; with the right affordances it can be accomplished with a small constant amount of data. If not for this already being deployed I personally believe the network would have already fallen into complete centeralization as a response to larger blocks: this was constructed and deployed in order to pull the network back from having a single pool with more than half the hashrate. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-05 20:16 ` Gregory Maxwell @ 2015-08-05 21:19 ` Arnoud Kouwenhoven - Pukaki Corp 2015-08-05 22:14 ` Gregory Maxwell 0 siblings, 1 reply; 14+ messages in thread From: Arnoud Kouwenhoven - Pukaki Corp @ 2015-08-05 21:19 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 3130 bytes --] Thanks for this (direct) feedback. It would make sense that if blocks can be submitted using ~5kb packets, that no further optimizations would be needed at this point. I will look into the relay network transmission protocol to understand how it works! I hear that you are saying that this network solves speed of transmission and thereby (technical) block size issues. Presumably it would solve speed of block validation too by prevalidating transactions. Assuming this is all true, and I have no reason to doubt that at this point, I do not understand why there is any discussion at all about the (technical) impact of large blocks, or why there are large numbers of miners building on invalid blocks (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or why there is any discussion about the speed of block validation (cpu processing time to verify blocks and transactions in blocks being a limitation). Our proposal aims at solving all three issues. Now I would be glad if the suggestions we made are already implemented, especially if that is in a more elegant approach. Great! Yet we still see all three discussions, which is a surprise if they have been solved. On Wed, Aug 5, 2015 at 2:16 PM, Gregory Maxwell <gmaxwell@gmail•com> wrote: > On Wed, Aug 5, 2015 at 7:53 PM, Arnoud Kouwenhoven - Pukaki Corp via > bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > > Thanks for the reply. My understanding is that the bitcoin relay network > is > > a backbone of connected high speed servers to increase the rate at which > > transactions and new blocks propagate - and remove a number of delays in > > processing. But it would still require the miners to download the entire > > block before building on top of it with any degree of confidence. > > Your understanding is outdated. > > The relay network includes an optimized transmission protocol which > enables sending the "entire" block typically in just a smal number of > bytes (much smaller than the summaries you suggest, which still leave > the participants needing to send the block). > > E.g. block 000ce90846 was 999950 bytes and the relay network protocol > sent it using at most 4906 bytes. > > No trust is required in this scheme because the entire block is > communicated using only a couple packets. > > The current scheme is highly simplified and its efficiency could be > increased greatly with small improvements, or if miners created blocks > in an aware manner.... but with a maximum size blocks turning into 5kb > with the current setup, there hardly appears to be a reason to do so > right now. > > Ultimately there is no need for information communicated with a block > at discovery time proportional to the size of the block; with the > right affordances it can be accomplished with a small constant amount > of data. > > If not for this already being deployed I personally believe the > network would have already fallen into complete centeralization as a > response to larger blocks: this was constructed and deployed in order > to pull the network back from having a single pool with more than half > the hashrate. > [-- Attachment #2: Type: text/html, Size: 3792 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-05 21:19 ` Arnoud Kouwenhoven - Pukaki Corp @ 2015-08-05 22:14 ` Gregory Maxwell 2015-08-06 17:16 ` Sergio Demian Lerner 0 siblings, 1 reply; 14+ messages in thread From: Gregory Maxwell @ 2015-08-05 22:14 UTC (permalink / raw) To: Arnoud Kouwenhoven - Pukaki Corp Cc: Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp <arnoud@pukaki•bz> wrote: > Thanks for this (direct) feedback. It would make sense that if blocks can be > submitted using ~5kb packets, that no further optimizations would be needed > at this point. I will look into the relay network transmission protocol to > understand how it works! > > I hear that you are saying that this network solves speed of transmission > and thereby (technical) block size issues. Presumably it would solve speed > of block validation too by prevalidating transactions. Correct. Bitcoin Core has cached validation for many years now... if not for that and other optimizations, things would be really broken right now. :) > Assuming this is all > true, and I have no reason to doubt that at this point, I do not understand > why there is any discussion at all about the (technical) impact of large > blocks, why there are large numbers of miners building on invalid blocks > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or why > there is any discussion about the speed of block validation (cpu processing > time to verify blocks and transactions in blocks being a limitation). I'm also mystified by a lot of the large block discussion, much of it is completely divorced from the technology as deployed; much less what we-- in industry-- know to be possible. I don't blame you or anyone in particular on this; it's a new area and we don't yet know what we need to know to know what we need to know; or to the extent that we do it hasn't had time to get effectively communicated. The technical/security implications of larger blocks are related to other things than propagation time, if you assume people are using the available efficient relay protocol (or better). SPV mining is a bit of a misnomer (If I coined the term, I'm sorry). What these parties are actually doing is blinding mining on top of other pools' stratum work. You can think of it as sub-pooling with hopping onto whatever pool has the highest block (I'll call it VFSSP in this post-- validation free stratum subpooling). It's very easy to implement, and there are other considerations. It was initially deployed at a time when a single pool in Europe has amassed more than half of the hashrate. This pool had propagation problems and a very high orphan rate, it may have (perhaps unintentionally) been performing a selfish mining attack; mining off their stratum work was an easy fix which massively cut down the orphan rates for anyone who did it. This was before the relay network protocol existed (the fact that all the hashpower was consolidating on a single pool was a major motivation for creating it). VFSSP also cuts through a number of practical issues miners have had: Miners that run their own bitcoin nodes in far away colocation (>100ms) due to local bandwidth or connectivity issues (censored internet); relay network hubs not being anywhere near by due to strange internet routing (e.g. japan to china going via the US for ... reasons...); the CreateNewBlock() function being very slow and unoptimized, etc. There are many other things like this-- and VFSSP avoids them causing delays even when you don't understand them or know about them. So even when they're easily fixed the VFSSP is a more general workaround. Mining operations are also usually operated in a largely fire and forget manner. There is a long history in (esp pooled) mining where someone sets up an operation and then hardly maintains it after the fact... so some of the use of VFSSP appears to just be inertia-- we have better solutions now, but they they work to deploy and changing things involves risk (which is heightened by a lack of good monitoring-- participants learn they are too latent by observing orphaned blocks at a cost of 25 BTC each). One of the frustrating things about incentives in this space is that bad outcomes are possible even when they're not necessary. E.g. if a miner can lower their orphan rate by deploying a new protocol (or simply fixing some faulty hardware in their infrastructure, like Bitcoin nodes running on cheap VPSes with remote storage) OR they can lower their orphan rate by pointing their hashpower at a free centeralized pool, they're likely to do the latter because it takes less effort. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-05 22:14 ` Gregory Maxwell @ 2015-08-06 17:16 ` Sergio Demian Lerner 2015-08-06 17:33 ` Olaoluwa Osuntokun ` (2 more replies) 0 siblings, 3 replies; 14+ messages in thread From: Sergio Demian Lerner @ 2015-08-06 17:16 UTC (permalink / raw) To: Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 4955 bytes --] Is there any up to date documentation about TheBlueMatt relay network including what kind of block compression it is currently doing? (apart from the source code) Regards, Sergio. On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev < bitcoin-dev@lists•linuxfoundation.org> wrote: > On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp > <arnoud@pukaki•bz> wrote: > > Thanks for this (direct) feedback. It would make sense that if blocks > can be > > submitted using ~5kb packets, that no further optimizations would be > needed > > at this point. I will look into the relay network transmission protocol > to > > understand how it works! > > > > I hear that you are saying that this network solves speed of transmission > > and thereby (technical) block size issues. Presumably it would solve > speed > > of block validation too by prevalidating transactions. > > > Correct. Bitcoin Core has cached validation for many years now... if > not for that and other optimizations, things would be really broken > right now. :) > > > Assuming this is all > > true, and I have no reason to doubt that at this point, I do not > understand > > why there is any discussion at all about the (technical) impact of large > > blocks, why there are large numbers of miners building on invalid blocks > > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or why > > there is any discussion about the speed of block validation (cpu > processing > > time to verify blocks and transactions in blocks being a limitation). > > I'm also mystified by a lot of the large block discussion, much of it > is completely divorced from the technology as deployed; much less what > we-- in industry-- know to be possible. I don't blame you or anyone in > particular on this; it's a new area and we don't yet know what we need > to know to know what we need to know; or to the extent that we do it > hasn't had time to get effectively communicated. > > The technical/security implications of larger blocks are related to > other things than propagation time, if you assume people are using the > available efficient relay protocol (or better). > > SPV mining is a bit of a misnomer (If I coined the term, I'm sorry). > What these parties are actually doing is blinding mining on top of > other pools' stratum work. You can think of it as sub-pooling with > hopping onto whatever pool has the highest block (I'll call it VFSSP > in this post-- validation free stratum subpooling). It's very easy to > implement, and there are other considerations. > > It was initially deployed at a time when a single pool in Europe has > amassed more than half of the hashrate. This pool had propagation > problems and a very high orphan rate, it may have (perhaps > unintentionally) been performing a selfish mining attack; mining off > their stratum work was an easy fix which massively cut down the orphan > rates for anyone who did it. This was before the relay network > protocol existed (the fact that all the hashpower was consolidating on > a single pool was a major motivation for creating it). > > VFSSP also cuts through a number of practical issues miners have had: > Miners that run their own bitcoin nodes in far away colocation > (>100ms) due to local bandwidth or connectivity issues (censored > internet); relay network hubs not being anywhere near by due to > strange internet routing (e.g. japan to china going via the US for ... > reasons...); the CreateNewBlock() function being very slow and > unoptimized, etc. There are many other things like this-- and VFSSP > avoids them causing delays even when you don't understand them or know > about them. So even when they're easily fixed the VFSSP is a more > general workaround. > > Mining operations are also usually operated in a largely fire and > forget manner. There is a long history in (esp pooled) mining where > someone sets up an operation and then hardly maintains it after the > fact... so some of the use of VFSSP appears to just be inertia-- we > have better solutions now, but they they work to deploy and changing > things involves risk (which is heightened by a lack of good > monitoring-- participants learn they are too latent by observing > orphaned blocks at a cost of 25 BTC each). > > One of the frustrating things about incentives in this space is that > bad outcomes are possible even when they're not necessary. E.g. if a > miner can lower their orphan rate by deploying a new protocol (or > simply fixing some faulty hardware in their infrastructure, like > Bitcoin nodes running on cheap VPSes with remote storage) OR they can > lower their orphan rate by pointing their hashpower at a free > centeralized pool, they're likely to do the latter because it takes > less effort. > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > [-- Attachment #2: Type: text/html, Size: 6065 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-06 17:16 ` Sergio Demian Lerner @ 2015-08-06 17:33 ` Olaoluwa Osuntokun 2015-08-06 18:17 ` Tom Harding 2015-08-06 20:38 ` Matt Corallo 2 siblings, 0 replies; 14+ messages in thread From: Olaoluwa Osuntokun @ 2015-08-06 17:33 UTC (permalink / raw) To: Sergio Demian Lerner, Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 5586 bytes --] Other than the source code, the best documentation I've come across is a few lines on IRC explaining the high-level design of the protocol: https://botbot.me/freenode/bitcoin-wizards/2015-07-10/?msg=44146764&page=2 On Thu, Aug 6, 2015 at 10:18 AM Sergio Demian Lerner via bitcoin-dev < bitcoin-dev@lists•linuxfoundation.org> wrote: > Is there any up to date documentation about TheBlueMatt relay network > including what kind of block compression it is currently doing? (apart from > the source code) > > Regards, Sergio. > > On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev < > bitcoin-dev@lists•linuxfoundation.org> wrote: > >> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp >> <arnoud@pukaki•bz> wrote: >> > Thanks for this (direct) feedback. It would make sense that if blocks >> can be >> > submitted using ~5kb packets, that no further optimizations would be >> needed >> > at this point. I will look into the relay network transmission protocol >> to >> > understand how it works! >> > >> > I hear that you are saying that this network solves speed of >> transmission >> > and thereby (technical) block size issues. Presumably it would solve >> speed >> > of block validation too by prevalidating transactions. >> >> >> Correct. Bitcoin Core has cached validation for many years now... if >> not for that and other optimizations, things would be really broken >> right now. :) >> >> > Assuming this is all >> > true, and I have no reason to doubt that at this point, I do not >> understand >> > why there is any discussion at all about the (technical) impact of large >> > blocks, why there are large numbers of miners building on invalid blocks >> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or >> why >> > there is any discussion about the speed of block validation (cpu >> processing >> > time to verify blocks and transactions in blocks being a limitation). >> >> I'm also mystified by a lot of the large block discussion, much of it >> is completely divorced from the technology as deployed; much less what >> we-- in industry-- know to be possible. I don't blame you or anyone in >> particular on this; it's a new area and we don't yet know what we need >> to know to know what we need to know; or to the extent that we do it >> hasn't had time to get effectively communicated. >> >> The technical/security implications of larger blocks are related to >> other things than propagation time, if you assume people are using the >> available efficient relay protocol (or better). >> >> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry). >> What these parties are actually doing is blinding mining on top of >> other pools' stratum work. You can think of it as sub-pooling with >> hopping onto whatever pool has the highest block (I'll call it VFSSP >> in this post-- validation free stratum subpooling). It's very easy to >> implement, and there are other considerations. >> >> It was initially deployed at a time when a single pool in Europe has >> amassed more than half of the hashrate. This pool had propagation >> problems and a very high orphan rate, it may have (perhaps >> unintentionally) been performing a selfish mining attack; mining off >> their stratum work was an easy fix which massively cut down the orphan >> rates for anyone who did it. This was before the relay network >> protocol existed (the fact that all the hashpower was consolidating on >> a single pool was a major motivation for creating it). >> >> VFSSP also cuts through a number of practical issues miners have had: >> Miners that run their own bitcoin nodes in far away colocation >> (>100ms) due to local bandwidth or connectivity issues (censored >> internet); relay network hubs not being anywhere near by due to >> strange internet routing (e.g. japan to china going via the US for ... >> reasons...); the CreateNewBlock() function being very slow and >> unoptimized, etc. There are many other things like this-- and VFSSP >> avoids them causing delays even when you don't understand them or know >> about them. So even when they're easily fixed the VFSSP is a more >> general workaround. >> >> Mining operations are also usually operated in a largely fire and >> forget manner. There is a long history in (esp pooled) mining where >> someone sets up an operation and then hardly maintains it after the >> fact... so some of the use of VFSSP appears to just be inertia-- we >> have better solutions now, but they they work to deploy and changing >> things involves risk (which is heightened by a lack of good >> monitoring-- participants learn they are too latent by observing >> orphaned blocks at a cost of 25 BTC each). >> >> One of the frustrating things about incentives in this space is that >> bad outcomes are possible even when they're not necessary. E.g. if a >> miner can lower their orphan rate by deploying a new protocol (or >> simply fixing some faulty hardware in their infrastructure, like >> Bitcoin nodes running on cheap VPSes with remote storage) OR they can >> lower their orphan rate by pointing their hashpower at a free >> centeralized pool, they're likely to do the latter because it takes >> less effort. >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > [-- Attachment #2: Type: text/html, Size: 7205 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-06 17:16 ` Sergio Demian Lerner 2015-08-06 17:33 ` Olaoluwa Osuntokun @ 2015-08-06 18:17 ` Tom Harding 2015-08-06 18:42 ` Gregory Maxwell 2015-08-06 20:55 ` Matt Corallo 2015-08-06 20:38 ` Matt Corallo 2 siblings, 2 replies; 14+ messages in thread From: Tom Harding @ 2015-08-06 18:17 UTC (permalink / raw) To: bitcoin-dev On 8/6/2015 10:16 AM, Sergio Demian Lerner via bitcoin-dev wrote: > Is there any up to date documentation about TheBlueMatt relay network > including what kind of block compression it is currently doing? (apart > from the source code) > Another question. Did the "relay network" relay 0000000000000000009cc829aa25b40b2cd4eb83dd498c12ad0d26d90c439d99, the BTC Nuggets block that was invalid post-softfork? If so, - Is there reason to believe that by so doing, it contributed to the growth of the 2015-07-04 fork? - Will the relay network at least validate block version numbers in the future? ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-06 18:17 ` Tom Harding @ 2015-08-06 18:42 ` Gregory Maxwell 2015-08-06 20:50 ` Matt Corallo 2015-08-06 20:55 ` Matt Corallo 1 sibling, 1 reply; 14+ messages in thread From: Gregory Maxwell @ 2015-08-06 18:42 UTC (permalink / raw) To: Tom Harding; +Cc: Bitcoin Dev On Thu, Aug 6, 2015 at 6:17 PM, Tom Harding via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > - Will the relay network at least validate block version numbers in the > future? It already validates block version numbers. It only relays valid transactions. Although, the block relaying itself is explicitly "unvalidated" and the software client can only usefully be used with a mempool maintaining full node (otherwise it doesn't provide much value, because the node must wait to validate the things). ... but that doesn't actually mean no validation at all is performed, many stateless checks are performed. On Thu, Aug 6, 2015 at 5:16 PM, Sergio Demian Lerner via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > Is there any up to date documentation about TheBlueMatt relay network > including what kind of block compression it is currently doing? (apart from > the source code) I don't know if Matt has an extensive writeup. But the basic optimization it performs is trivial. I wouldn't call it compression, though it does have some analog to RTP "header compression". All it does is relay transactions verified by a local node and keeps a FIFO of the relayed transactions in both directions, which is synchronous on each side. When a block is recieved on either side, it replaces transactions with their indexes in the FIFO and relays it along. Transactions not in the fifo are escaped and sent whole. On the other side the block is reconstructed using the stored data and handed to the node (where the preforwarded transactions would have also been pre-validated). There is some more than basic elaboration for resource management (e.g. multiple queues for different transaction sizes)-- and more recently using block templates to learn transaction priority be a bit more immune to spam attacks, but its fairly simple. Much better could be done about intelligently managing the queues or efficiently transmitting the membership sets, etc. It's just basically the simplest thing that isn't completely stupid. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-06 18:42 ` Gregory Maxwell @ 2015-08-06 20:50 ` Matt Corallo 0 siblings, 0 replies; 14+ messages in thread From: Matt Corallo @ 2015-08-06 20:50 UTC (permalink / raw) To: Gregory Maxwell, Gregory Maxwell via bitcoin-dev, Tom Harding; +Cc: Bitcoin Dev On August 6, 2015 8:42:38 PM GMT+02:00, Gregory Maxwell via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: >On Thu, Aug 6, 2015 at 6:17 PM, Tom Harding via bitcoin-dev ><bitcoin-dev@lists•linuxfoundation.org> wrote: >> - Will the relay network at least validate block version numbers in >the >> future? > >It already validates block version numbers. > >It only relays valid transactions. > >Although, the block relaying itself is explicitly "unvalidated" and >the software client can only usefully be used with a mempool >maintaining full node (otherwise it doesn't provide much value, >because the node must wait to validate the things). ... but that >doesn't actually mean no validation at all is performed, many >stateless checks are performed. > >On Thu, Aug 6, 2015 at 5:16 PM, Sergio Demian Lerner via bitcoin-dev ><bitcoin-dev@lists•linuxfoundation.org> wrote: >> Is there any up to date documentation about TheBlueMatt relay network >> including what kind of block compression it is currently doing? >(apart from >> the source code) > >I don't know if Matt has an extensive writeup. But the basic >optimization it performs is trivial. I wouldn't call it compression, >though it does have some analog to RTP "header compression". > >All it does is relay transactions verified by a local node and keeps a >FIFO of the relayed transactions in both directions, which is >synchronous on each side. > >When a block is recieved on either side, it replaces transactions with >their indexes in the FIFO and relays it along. Transactions not in the >fifo are escaped and sent whole. On the other side the block is >reconstructed using the stored data and handed to the node (where the >preforwarded transactions would have also been pre-validated). > >There is some more than basic elaboration for resource management >(e.g. multiple queues for different transaction sizes)-- and more No, just one queue, but it has a count-of-oversize-txn-limit, in addition to a size. >recently using block templates to learn transaction priority be a bit >more immune to spam attacks, but its fairly simple. Except it doesn't really work :( (see https://github.com/TheBlueMatt/RelayNode/issues/12#issuecomment-128234446) >Much better could be done about intelligently managing the queues or >efficiently transmitting the membership sets, etc. It's just >basically the simplest thing that isn't completely stupid. Patches welcome :) (read the issues list first... Rewriting the protocol from scratch is by far not the biggest win here). Matt ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-06 18:17 ` Tom Harding 2015-08-06 18:42 ` Gregory Maxwell @ 2015-08-06 20:55 ` Matt Corallo 1 sibling, 0 replies; 14+ messages in thread From: Matt Corallo @ 2015-08-06 20:55 UTC (permalink / raw) To: Tom Harding, Tom Harding via bitcoin-dev, bitcoin-dev On August 6, 2015 8:17:35 PM GMT+02:00, Tom Harding via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: >On 8/6/2015 10:16 AM, Sergio Demian Lerner via bitcoin-dev wrote: >> Is there any up to date documentation about TheBlueMatt relay network >> including what kind of block compression it is currently doing? >(apart >> from the source code) >> > >Another question. > >Did the "relay network" relay >0000000000000000009cc829aa25b40b2cd4eb83dd498c12ad0d26d90c439d99, the >BTC Nuggets block that was invalid post-softfork? If so, The version check was only added hours after the initial fork, so it should have (assuming BTC Nuggets or anyone who accepted it is running a client) > - Is there reason to believe that by so doing, it contributed to the >growth of the 2015-07-04 fork? The reason other miners mined on that fork is because they were watching each other's stratum servers, so the relay network should not have had a significant effect. Still, even in a different fork, miners already aggressively relay around the network/between each other, so I'm not so worried. >- Will the relay network at least validate block version numbers in the >future? > >_______________________________________________ >bitcoin-dev mailing list >bitcoin-dev@lists•linuxfoundation.org >https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-06 17:16 ` Sergio Demian Lerner 2015-08-06 17:33 ` Olaoluwa Osuntokun 2015-08-06 18:17 ` Tom Harding @ 2015-08-06 20:38 ` Matt Corallo 2 siblings, 0 replies; 14+ messages in thread From: Matt Corallo @ 2015-08-06 20:38 UTC (permalink / raw) To: Sergio Demian Lerner, Sergio Demian Lerner via bitcoin-dev, Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 5827 bytes --] No, don't think so, the protocol is, essentially, relay transactions, when you get a block, send header, iterate over transactions, for each, either use two bytes for nth-recent-transaction-relayed, use 0xffff-3-byte-length-transaction-data. There are quite a few implementation details, and lots of things could be improved, but that is pretty much how it works. Matt On August 6, 2015 7:16:56 PM GMT+02:00, Sergio Demian Lerner via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: >Is there any up to date documentation about TheBlueMatt relay network >including what kind of block compression it is currently doing? (apart >from >the source code) > >Regards, Sergio. > >On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev < >bitcoin-dev@lists•linuxfoundation.org> wrote: > >> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp >> <arnoud@pukaki•bz> wrote: >> > Thanks for this (direct) feedback. It would make sense that if >blocks >> can be >> > submitted using ~5kb packets, that no further optimizations would >be >> needed >> > at this point. I will look into the relay network transmission >protocol >> to >> > understand how it works! >> > >> > I hear that you are saying that this network solves speed of >transmission >> > and thereby (technical) block size issues. Presumably it would >solve >> speed >> > of block validation too by prevalidating transactions. >> >> >> Correct. Bitcoin Core has cached validation for many years now... if >> not for that and other optimizations, things would be really broken >> right now. :) >> >> > Assuming this is all >> > true, and I have no reason to doubt that at this point, I do not >> understand >> > why there is any discussion at all about the (technical) impact of >large >> > blocks, why there are large numbers of miners building on invalid >blocks >> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), >or why >> > there is any discussion about the speed of block validation (cpu >> processing >> > time to verify blocks and transactions in blocks being a >limitation). >> >> I'm also mystified by a lot of the large block discussion, much of it >> is completely divorced from the technology as deployed; much less >what >> we-- in industry-- know to be possible. I don't blame you or anyone >in >> particular on this; it's a new area and we don't yet know what we >need >> to know to know what we need to know; or to the extent that we do it >> hasn't had time to get effectively communicated. >> >> The technical/security implications of larger blocks are related to >> other things than propagation time, if you assume people are using >the >> available efficient relay protocol (or better). >> >> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry). >> What these parties are actually doing is blinding mining on top of >> other pools' stratum work. You can think of it as sub-pooling with >> hopping onto whatever pool has the highest block (I'll call it VFSSP >> in this post-- validation free stratum subpooling). It's very easy >to >> implement, and there are other considerations. >> >> It was initially deployed at a time when a single pool in Europe has >> amassed more than half of the hashrate. This pool had propagation >> problems and a very high orphan rate, it may have (perhaps >> unintentionally) been performing a selfish mining attack; mining off >> their stratum work was an easy fix which massively cut down the >orphan >> rates for anyone who did it. This was before the relay network >> protocol existed (the fact that all the hashpower was consolidating >on >> a single pool was a major motivation for creating it). >> >> VFSSP also cuts through a number of practical issues miners have had: >> Miners that run their own bitcoin nodes in far away colocation >> (>100ms) due to local bandwidth or connectivity issues (censored >> internet); relay network hubs not being anywhere near by due to >> strange internet routing (e.g. japan to china going via the US for >... >> reasons...); the CreateNewBlock() function being very slow and >> unoptimized, etc. There are many other things like this-- and VFSSP >> avoids them causing delays even when you don't understand them or >know >> about them. So even when they're easily fixed the VFSSP is a more >> general workaround. >> >> Mining operations are also usually operated in a largely fire and >> forget manner. There is a long history in (esp pooled) mining where >> someone sets up an operation and then hardly maintains it after the >> fact... so some of the use of VFSSP appears to just be inertia-- we >> have better solutions now, but they they work to deploy and changing >> things involves risk (which is heightened by a lack of good >> monitoring-- participants learn they are too latent by observing >> orphaned blocks at a cost of 25 BTC each). >> >> One of the frustrating things about incentives in this space is that >> bad outcomes are possible even when they're not necessary. E.g. if a >> miner can lower their orphan rate by deploying a new protocol (or >> simply fixing some faulty hardware in their infrastructure, like >> Bitcoin nodes running on cheap VPSes with remote storage) OR they >can >> lower their orphan rate by pointing their hashpower at a free >> centeralized pool, they're likely to do the latter because it takes >> less effort. >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists•linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> > > >------------------------------------------------------------------------ > >_______________________________________________ >bitcoin-dev mailing list >bitcoin-dev@lists•linuxfoundation.org >https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [-- Attachment #2: Type: text/html, Size: 7226 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation 2015-08-05 19:07 [bitcoin-dev] Idea: Efficient bitcoin block propagation Arnoud Kouwenhoven - Pukaki Corp 2015-08-05 19:27 ` Matt Corallo @ 2015-08-07 7:14 ` jl2012 1 sibling, 0 replies; 14+ messages in thread From: jl2012 @ 2015-08-07 7:14 UTC (permalink / raw) To: Arnoud Kouwenhoven - Pukaki Corp; +Cc: bitcoin-dev Your proposal fails here: "If the block defined in the Guarantee Message has not been shown" What is blockchain? You can see blockchain as a mechanism to prove something has been shown by certain order. Therefore, it is not possible to prove something has not been shown with blockchain. Your proposal works only with a centralized trusted party. Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev 於 2015-08-05 15:07 寫到: > Hello all. > > We’d like to share an idea we have to dramatically increase the > bitcoin block propagation speed after a new block has been mined for > the first time. ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2015-08-07 7:15 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-08-05 19:07 [bitcoin-dev] Idea: Efficient bitcoin block propagation Arnoud Kouwenhoven - Pukaki Corp 2015-08-05 19:27 ` Matt Corallo 2015-08-05 19:53 ` Arnoud Kouwenhoven - Pukaki Corp 2015-08-05 20:16 ` Gregory Maxwell 2015-08-05 21:19 ` Arnoud Kouwenhoven - Pukaki Corp 2015-08-05 22:14 ` Gregory Maxwell 2015-08-06 17:16 ` Sergio Demian Lerner 2015-08-06 17:33 ` Olaoluwa Osuntokun 2015-08-06 18:17 ` Tom Harding 2015-08-06 18:42 ` Gregory Maxwell 2015-08-06 20:50 ` Matt Corallo 2015-08-06 20:55 ` Matt Corallo 2015-08-06 20:38 ` Matt Corallo 2015-08-07 7:14 ` jl2012
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox