* [Bitcoin-development] Decentralizing mining [not found] ` <20130531165445.GA29104@petertodd.org> @ 2013-05-31 16:57 ` Peter Todd 2013-05-31 18:14 ` Adam Back 2013-06-10 21:09 ` Peter Todd 0 siblings, 2 replies; 9+ messages in thread From: Peter Todd @ 2013-05-31 16:57 UTC (permalink / raw) To: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 2841 bytes --] I just posted the following to bitcointalk. https://bitcointalk.org/index.php?topic=221164.0 Right now between two to four running the largest pools control Bitcoin in the short term. That's a lot of hashing power in the hands of very, very few people. In addition pools have little incentive to run secure operations, and many pools have been hacked with their funds stolen. Those hacks could just have easily been used to attack the network itself. This needs to change. Pooled-solo mining is a concept Gregory Maxwell, Luke Dashjr and I were discussing at the conference two weeks ago. (credit goes to Greg and Luke; I was mostly just listening) Basically the idea is that miners with mining equipment run a local Bitcoin node and use that node to construct the blocks they mine - the same as if they were solo mining. The pools job is then to only track shares and organize payouts. If the pool gets hacked the worst that can happen is miners are ripped off, rather than Bitcoin itself being attacked. With pooled-solo mining even a pool with a majority of hashing power wouldn't be able to do much harm to Bitcoin. (depending on the implementation they may be able to blacklist specific transactions - the pool needs to know what transactions are in the share to credit fees properly) Tech-wise Luke created getblocktemplate last year as a means to audit mining pools. I'm sure Greg and Luke can explain the nitty gritty details better than I can, but essentially the plan is to take getblocktemplate and extend it as required for pooled-solo mining. This will include pool and miner software initially, and later improvements to GUIs and what not to make the whole process easier. With the success of my recent video project I also want to make this Keep Bitcoin Free's next project, specifically funding a developer (likely Luke) to make this happen. Additionally once software is written and easily usable a good follow-up would be a video and other media to promote the idea to miners. No guarantees we'll be able to come up with commercially competitive remuneration, but we can at least come up with a "Thank you" tip. But first lets discuss the technical requirements to get an idea of what the scope is. Finally, for the record, a big part of the reason why myself and other Keep Bitcoin Free supporters are interested in doing this is very much to take power over the direction of the network from big pools and put it into the hands of thousands of individual miners. It's much easier to convince people that changes to Bitcoin, like increasing the blocksize, are directly impacting decentralization when individual miners are seeing that happen to themselves. -- 'peter'[:-1]@petertodd.org 00000000000000c14fa7031b2431ab32785efdf1e5aaecc83555ee52a2fc550b [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining 2013-05-31 16:57 ` [Bitcoin-development] Decentralizing mining Peter Todd @ 2013-05-31 18:14 ` Adam Back 2013-06-10 21:09 ` Peter Todd 1 sibling, 0 replies; 9+ messages in thread From: Adam Back @ 2013-05-31 18:14 UTC (permalink / raw) To: Peter Todd; +Cc: bitcoin-development I like this idea a lot. To add: I think it is a bug and security risk if pooled-solo or (current pooled miners) do not add randomness to their extraNonce2 (like 128-bits of it). For privacy and to avoid various hostile-pre-mining attacks it should be done this way. Lack of the self-chosen challenge field is the reason Satoshi's first year mining is marked (plus forgetting to reset the counter). (Bitcoind I believe considered the direct miners key as defense enough as a stand in for self-chosen challenge, which has a few problems). The base counter I think is only 32-bits, the extranonce2 itself being random can be incremented while still looking random. But incrementing extranonce directy while initializing it to 0 is not good (per previous mining extranone marked coins bug - is that even fixed?) (You dont want to reveal the miners power in his pool shares, if the full counter is revealed with no randomness it also reveals how many iterations he can do since the block start). Adam On Fri, May 31, 2013 at 12:57:58PM -0400, Peter Todd wrote: >I just posted the following to bitcointalk. > >https://bitcointalk.org/index.php?topic=221164.0 > > >Right now between two to four running the largest pools control Bitcoin >in the short term. That's a lot of hashing power in the hands of very, >very few people. In addition pools have little incentive to run secure >operations, and many pools have been hacked with their funds stolen. >Those hacks could just have easily been used to attack the network >itself. > >This needs to change. > >Pooled-solo mining is a concept Gregory Maxwell, Luke Dashjr and I were >discussing at the conference two weeks ago. (credit goes to Greg and >Luke; I was mostly just listening) Basically the idea is that miners >with mining equipment run a local Bitcoin node and use that node to >construct the blocks they mine - the same as if they were solo mining. >The pools job is then to only track shares and organize payouts. > >If the pool gets hacked the worst that can happen is miners are ripped >off, rather than Bitcoin itself being attacked. With pooled-solo mining >even a pool with a majority of hashing power wouldn't be able to do much >harm to Bitcoin. (depending on the implementation they may be able to >blacklist specific transactions - the pool needs to know what >transactions are in the share to credit fees properly) > >Tech-wise Luke created getblocktemplate last year as a means to audit >mining pools. I'm sure Greg and Luke can explain the nitty gritty >details better than I can, but essentially the plan is to take >getblocktemplate and extend it as required for pooled-solo mining. This >will include pool and miner software initially, and later improvements >to GUIs and what not to make the whole process easier. > > >With the success of my recent video project I also want to make this >Keep Bitcoin Free's next project, specifically funding a developer >(likely Luke) to make this happen. Additionally once software is written >and easily usable a good follow-up would be a video and other media to >promote the idea to miners. No guarantees we'll be able to come up with >commercially competitive remuneration, but we can at least come up with >a "Thank you" tip. But first lets discuss the technical requirements to >get an idea of what the scope is. > > >Finally, for the record, a big part of the reason why myself and other >Keep Bitcoin Free supporters are interested in doing this is very much >to take power over the direction of the network from big pools and put >it into the hands of thousands of individual miners. It's much easier to >convince people that changes to Bitcoin, like increasing the blocksize, >are directly impacting decentralization when individual miners are >seeing that happen to themselves. > >-- >'peter'[:-1]@petertodd.org >00000000000000c14fa7031b2431ab32785efdf1e5aaecc83555ee52a2fc550b >------------------------------------------------------------------------------ >Get 100% visibility into Java/.NET code with AppDynamics Lite >It's a free troubleshooting tool designed for production >Get down to code-level detail for bottlenecks, with <2% overhead. >Download for free and get started troubleshooting in minutes. >http://p.sf.net/sfu/appdyn_d2d_ap2 >_______________________________________________ >Bitcoin-development mailing list >Bitcoin-development@lists•sourceforge.net >https://lists.sourceforge.net/lists/listinfo/bitcoin-development ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining 2013-05-31 16:57 ` [Bitcoin-development] Decentralizing mining Peter Todd 2013-05-31 18:14 ` Adam Back @ 2013-06-10 21:09 ` Peter Todd 2013-06-10 21:23 ` Luke-Jr 2013-06-10 21:31 ` Melvin Carvalho 1 sibling, 2 replies; 9+ messages in thread From: Peter Todd @ 2013-06-10 21:09 UTC (permalink / raw) To: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 3319 bytes --] So here's the parts that need to be done for step #1: # Protocol Work Basic idea is the miner makes two connections, their pool, and a local bitcoind. They always (usually?) work on the subset of transactions common to both the pool's getblocktemplate and their local one. When they find a share if it doesn't meet difficulty they just hand it to the pool. Currently that is done by handing the whole block over, correct? I know the BIP says otherwise, but we should optimize this to just hand over tx hashes where possible. If the share does meet difficulty, hand it to both the pool and the local bitcoind. Should hand it to the pool first though, because the pool likely has the fastest block propagation, then hand it to local bitcoind. An optimized version may want to have some record of measured bandwidth - this applies Bitcoin in general too, although also has other issues. ## Reducing bandwidth How about for normal shares we just pass the block header, and have the pool randomly pick a subset of transactions to audit? Any fraud cancels the users shares. This will work best in conjunction with a UTXO proof tree to prove fees, or by just picking whole shares randomly to audit. We'll need persistent share storage so if your connection disconnects you can provide the pool with the full share later though. Incedentally, note how the miner can do the reverse: pick random block headers and challenge the pool to prove that they correspond to a valid block. With some modifications Stratum can support this approach. ## Delibrate orphaning of slow to propagate blocks Block headers can be flooded-filled much faster than blocks themselves. They are also small enough to fit into a UDP packet. Nodes should pass headers around separately via UDP, optinally with some tiny number of transactions. When we see a valid block header whose contents we do not know about a miner should switch to mining empty or near empty blocks in solo mode that would orphan the still propagating block. Doing this is safe, we'll never build on an invalid block, economically rational while the inflation subsidy is still high, and helps reduce (although not eliminate!) the advantage a large miner with high-bandwidth connections has over those who don't. Of course, the other option is to build a block extending the one you don't know about, which is even more rational, but doing poses major risks to Bitcoin... This functionality can be implemented later - it's not strictly part of pooled-solo mode. # Pool work So does eliopool already accept arbitrary shares like this and do the correct accounting already? (IE adjust share amount based on fees?) What happens when the pool doesn't get the share directly, but does see the new block? + possible protocol extensions # Miner work Basically we need code to merge the two block templates together to find commonality. I guess you probably want to implement this in bfgminer first, so add the code to libblkmaker first, then maybe python-blkmaker. We also want an automatic fallback to local solo mining if the pool can't be contacted. + possible protocol extensions -- 'peter'[:-1]@petertodd.org 000000000000005576673e616271f762a5d8779d5fe7796c6e43ed43df5aa189 [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining 2013-06-10 21:09 ` Peter Todd @ 2013-06-10 21:23 ` Luke-Jr 2013-06-14 20:06 ` Peter Todd 2013-06-10 21:31 ` Melvin Carvalho 1 sibling, 1 reply; 9+ messages in thread From: Luke-Jr @ 2013-06-10 21:23 UTC (permalink / raw) To: bitcoin-development On Monday, June 10, 2013 9:09:13 PM Peter Todd wrote: > # Protocol Work This is basically done. > Basic idea is the miner makes two connections, their pool, and a local > bitcoind. > > They always (usually?) work on the subset of transactions common to both > the pool's getblocktemplate and their local one. When they find a share > if it doesn't meet difficulty they just hand it to the pool. Currently > that is done by handing the whole block over, correct? I know the BIP > says otherwise, but we should optimize this to just hand over tx hashes > where possible. The plan was to tell the pool it doesn't need to send transactions at all, and only work on the ones from bitcoind. Currently, share submissions are just the block header and coinbase transaction; in this case, however, the miner will need to send merkle links also, possibly just once via a block proposal in advance. > If the share does meet difficulty, hand it to both the pool and the > local bitcoind. Should hand it to the pool first though, because the > pool likely has the fastest block propagation, then hand it to local > bitcoind. An optimized version may want to have some record of measured > bandwidth - this applies Bitcoin in general too, although also has other > issues. Currently, BFGMiner is doing submission to the pool, waiting for a response, then submitting to a local bitcoind. This is because the pool might need to receive/record the share before it processes the block on bitcoind, or you could lose credit for it. The response from the pool is rather small (a single TCP packet), so this shouldn't delay much longer. > ## Reducing bandwidth > > How about for normal shares we just pass the block header, and have the > pool randomly pick a subset of transactions to audit? Any fraud cancels > the users shares. This will work best in conjunction with a UTXO proof > tree to prove fees, or by just picking whole shares randomly to audit. Might as well just use higher difficulty shares (each one audited) for the same effect. Block proposals allow the miner to tell the pool its transaction set once (per txset change) for any number of shares. IF bandwidth becomes a real problem, I have a draft of a "GBT 2.0" that does some more improvement in this area. > # Pool work > > So does eliopool already accept arbitrary shares like this and do the > correct accounting already? (IE adjust share amount based on fees?) What > happens when the pool doesn't get the share directly, but does see the > new block? > > + possible protocol extensions I don't follow. > # Miner work > > Basically we need code to merge the two block templates together to find > commonality. I guess you probably want to implement this in bfgminer > first, so add the code to libblkmaker first, then maybe python-blkmaker. libblkmaker's API was designed for this from the start, so it should be fairly easily implemented. > We also want an automatic fallback to local solo mining if the pool > can't be contacted. > > + possible protocol extensions Failover already functions, but probably could use a rewrite... Luke ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining 2013-06-10 21:23 ` Luke-Jr @ 2013-06-14 20:06 ` Peter Todd 2013-06-14 21:05 ` Luke-Jr 2013-06-17 15:16 ` Jeff Garzik 0 siblings, 2 replies; 9+ messages in thread From: Peter Todd @ 2013-06-14 20:06 UTC (permalink / raw) To: Luke-Jr; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 5609 bytes --] On Mon, Jun 10, 2013 at 09:23:14PM +0000, Luke-Jr wrote: > > They always (usually?) work on the subset of transactions common to both > > the pool's getblocktemplate and their local one. When they find a share > > if it doesn't meet difficulty they just hand it to the pool. Currently > > that is done by handing the whole block over, correct? I know the BIP > > says otherwise, but we should optimize this to just hand over tx hashes > > where possible. > > The plan was to tell the pool it doesn't need to send transactions at all, and > only work on the ones from bitcoind. Currently, share submissions are just the > block header and coinbase transaction; in this case, however, the miner will > need to send merkle links also, possibly just once via a block proposal in > advance. It strikes me that this would work best if the pool has a mempool with child-pays-for-parent support where conflicts *are* allowed. IE you record whatever transactions you know about, conflicting or not, calculate which ones gives you the most fees/kb, and then figure out which set of non-conflicting ones are optimal. Of course, "optimal" is the knapsack problem... Now you can easilly tell the miners working on shares for you which tx's would be optimal if they wish to know, and at the same time whatever shares they send you are most likely to include transactions in your mempool inventory, and thus they can send just tx hashes to reduce bandwidth. Part of the broader issue that when we send peers INV advertisements we should be telling them what the fee/kb is so our peers can prioritize properly. That'd also help for the case where you want to broadcast two transactions in a row, but the pair is only profitable because the second is paying the fee for the first. Speaking of, the way we tell peers about new blocks is really suboptimal: we tell every peer, in no particular order, about a new block via a block INV message, and then we give them the new block in parallel. I was looking through comp-sci papers on optimal flood-fill/gossip algorithms for random graph networks and it appears that optimal is to spend all your bandwidth to send the message to your fastest peer first, followed by your next fastest and so on. This works best because you get the exponential growth scaling faster by propagating the message as "deep" as possible in the network, and it then can flood outwards from there. Just sorting the peer list by #inv-recevied/time when doing INV pushes and when attending to incoming messages would probably be a big improvement. > > If the share does meet difficulty, hand it to both the pool and the > > local bitcoind. Should hand it to the pool first though, because the > > pool likely has the fastest block propagation, then hand it to local > > bitcoind. An optimized version may want to have some record of measured > > bandwidth - this applies Bitcoin in general too, although also has other > > issues. > > Currently, BFGMiner is doing submission to the pool, waiting for a response, > then submitting to a local bitcoind. This is because the pool might need to > receive/record the share before it processes the block on bitcoind, or you > could lose credit for it. The response from the pool is rather small (a single > TCP packet), so this shouldn't delay much longer. Right, I guess the pool wants to be sure you were actually the one who found the share, rather than just someone who was lucky enough to see it on the network and submitted it as your own. > > ## Reducing bandwidth > > > > How about for normal shares we just pass the block header, and have the > > pool randomly pick a subset of transactions to audit? Any fraud cancels > > the users shares. This will work best in conjunction with a UTXO proof > > tree to prove fees, or by just picking whole shares randomly to audit. > > Might as well just use higher difficulty shares (each one audited) for the > same effect. Block proposals allow the miner to tell the pool its transaction > set once (per txset change) for any number of shares. That's a good point - the current practice most pools seem to follow of about a share per second seems very excessive to me. On the other hand, it does have good user optics. The right solution might be something akin to P2Pool where the UI level is telling the user shares are being found so it's clear "stuff is happening", but under the hood only a small subset are ever sent to the pool. > > # Pool work > > > > So does eliopool already accept arbitrary shares like this and do the > > correct accounting already? (IE adjust share amount based on fees?) What > > happens when the pool doesn't get the share directly, but does see the > > new block? > > > > + possible protocol extensions > > I don't follow. What part don't you follow? > > # Miner work > > > > Basically we need code to merge the two block templates together to find > > commonality. I guess you probably want to implement this in bfgminer > > first, so add the code to libblkmaker first, then maybe python-blkmaker. > > libblkmaker's API was designed for this from the start, so it should be fairly > easily implemented. Good. > > We also want an automatic fallback to local solo mining if the pool > > can't be contacted. > > > > + possible protocol extensions > > Failover already functions, but probably could use a rewrite... Sounds good. -- 'peter'[:-1]@petertodd.org 000000000000006ed83d97e7266a66dcb057cd53750795bd209076fa0a34d4ec [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining 2013-06-14 20:06 ` Peter Todd @ 2013-06-14 21:05 ` Luke-Jr 2013-06-17 15:16 ` Jeff Garzik 1 sibling, 0 replies; 9+ messages in thread From: Luke-Jr @ 2013-06-14 21:05 UTC (permalink / raw) To: Peter Todd; +Cc: bitcoin-development On Friday, June 14, 2013 8:06:54 PM Peter Todd wrote: > On Mon, Jun 10, 2013 at 09:23:14PM +0000, Luke-Jr wrote: > > Might as well just use higher difficulty shares (each one audited) for > > the same effect. Block proposals allow the miner to tell the pool its > > transaction set once (per txset change) for any number of shares. > > That's a good point - the current practice most pools seem to follow of > about a share per second seems very excessive to me. On the other hand, > it does have good user optics. The right solution might be something > akin to P2Pool where the UI level is telling the user shares are being > found so it's clear "stuff is happening", but under the hood only a > small subset are ever sent to the pool. Share rate is relevant to more than user information - it also affects the variance of reward/payout. I disagree with claiming shares are found when they're not sent to the pool - this makes auditing and troubleshooting more difficult; for example, see the GUIMiner bug where it reports shares despite misinterpreting the pool's target and submitting nothing at all (this happens when the pool uses pdiff 1). > > > # Pool work > > > > > > So does eliopool already accept arbitrary shares like this and do the > > > correct accounting already? (IE adjust share amount based on fees?) > > > What happens when the pool doesn't get the share directly, but does > > > see the new block? > > > > > > + possible protocol extensions > > > > I don't follow. > > What part don't you follow? I don't understand the first two questions here at all. Luke ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining 2013-06-14 20:06 ` Peter Todd 2013-06-14 21:05 ` Luke-Jr @ 2013-06-17 15:16 ` Jeff Garzik 2013-06-17 17:39 ` Peter Todd 1 sibling, 1 reply; 9+ messages in thread From: Jeff Garzik @ 2013-06-17 15:16 UTC (permalink / raw) To: Peter Todd; +Cc: bitcoin-development On Fri, Jun 14, 2013 at 4:06 PM, Peter Todd <pete@petertodd•org> wrote: > It strikes me that this would work best if the pool has a mempool with > child-pays-for-parent support where conflicts *are* allowed. > > IE you record whatever transactions you know about, conflicting or not, > calculate which ones gives you the most fees/kb, and then figure out > which set of non-conflicting ones are optimal. Of course, "optimal" is > the knapsack problem... > > Now you can easilly tell the miners working on shares for you which tx's > would be optimal if they wish to know, and at the same time whatever > shares they send you are most likely to include transactions in your > mempool inventory, and thus they can send just tx hashes to reduce > bandwidth. > > > Part of the broader issue that when we send peers INV advertisements we > should be telling them what the fee/kb is so our peers can prioritize > properly. That'd also help for the case where you want to broadcast two > transactions in a row, but the pair is only profitable because the > second is paying the fee for the first. Interesting proposals, particularly this last. The net result impact is, however, that which was criticized in at least one forum thread: replace-with-higher-fee. > Speaking of, the way we tell peers about new blocks is really > suboptimal: we tell every peer, in no particular order, about a new > block via a block INV message, and then we give them the new block in > parallel. I was looking through comp-sci papers on optimal > flood-fill/gossip algorithms for random graph networks and it appears > that optimal is to spend all your bandwidth to send the message to your > fastest peer first, followed by your next fastest and so on. This works > best because you get the exponential growth scaling faster by > propagating the message as "deep" as possible in the network, and it > then can flood outwards from there. Just sorting the peer list by > #inv-recevied/time when doing INV pushes and when attending to incoming > messages would probably be a big improvement. In terms of packet size, I would like to look into the network-wide costs of simply broadcasting block header + coinbase TX + TX list. I bet miners would love to opt into that. >> > If the share does meet difficulty, hand it to both the pool and the >> > local bitcoind. Should hand it to the pool first though, because the >> > pool likely has the fastest block propagation, then hand it to local >> > bitcoind. An optimized version may want to have some record of measured >> > bandwidth - this applies Bitcoin in general too, although also has other >> > issues. >> >> Currently, BFGMiner is doing submission to the pool, waiting for a response, >> then submitting to a local bitcoind. This is because the pool might need to >> receive/record the share before it processes the block on bitcoind, or you >> could lose credit for it. The response from the pool is rather small (a single >> TCP packet), so this shouldn't delay much longer. > > Right, I guess the pool wants to be sure you were actually the one who > found the share, rather than just someone who was lucky enough to see it > on the network and submitted it as your own. > >> > ## Reducing bandwidth >> > >> > How about for normal shares we just pass the block header, and have the >> > pool randomly pick a subset of transactions to audit? Any fraud cancels >> > the users shares. This will work best in conjunction with a UTXO proof >> > tree to prove fees, or by just picking whole shares randomly to audit. >> >> Might as well just use higher difficulty shares (each one audited) for the >> same effect. Block proposals allow the miner to tell the pool its transaction >> set once (per txset change) for any number of shares. > > That's a good point - the current practice most pools seem to follow of > about a share per second seems very excessive to me. On the other hand, > it does have good user optics. The right solution might be something > akin to P2Pool where the UI level is telling the user shares are being > found so it's clear "stuff is happening", but under the hood only a > small subset are ever sent to the pool. With the onslaught of ASIC mining, most big pools are past a share per second. Variable difficulty or set-to-higher-difficulty quickly became the norm, almost out of necessity. Personally, I think most pools should target at _least_ 5-10 seconds per share, no matter the strength of the miner. -- Jeff Garzik Senior Software Engineer and open source evangelist BitPay, Inc. https://bitpay.com/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining 2013-06-17 15:16 ` Jeff Garzik @ 2013-06-17 17:39 ` Peter Todd 0 siblings, 0 replies; 9+ messages in thread From: Peter Todd @ 2013-06-17 17:39 UTC (permalink / raw) To: Jeff Garzik; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 3564 bytes --] On Mon, Jun 17, 2013 at 11:16:01AM -0400, Jeff Garzik wrote: > > Part of the broader issue that when we send peers INV advertisements we > > should be telling them what the fee/kb is so our peers can prioritize > > properly. That'd also help for the case where you want to broadcast two > > transactions in a row, but the pair is only profitable because the > > second is paying the fee for the first. > > Interesting proposals, particularly this last. The net result impact > is, however, that which was criticized in at least one forum thread: > replace-with-higher-fee. Actually the two are orthogonal: a low-priority no-fee tx might result because it was from a customer paying a merchant via the payment protocol. The merchant can then respend that tx with a fee to cover both, but with the current mempool arrangement if the no-fee tx load is high actually getting that first tx to propagate so the second can will be difficult. A nice way to do this would be to accept tx's into your mempool indiscriminately but delay broadcasting INV messages until you find child tx's that make the low-profit ones worth mining. When you do find a child with a sufficiently high fee, send an INVGROUP message to notify your peers of the new opportunity. Different nodes will have different ideas of what priority TX deserves to be broadcast, but here provided the group meets the threshold a peer will always find out. > > Speaking of, the way we tell peers about new blocks is really > > suboptimal: we tell every peer, in no particular order, about a new > > block via a block INV message, and then we give them the new block in > > parallel. I was looking through comp-sci papers on optimal > > flood-fill/gossip algorithms for random graph networks and it appears > > that optimal is to spend all your bandwidth to send the message to your > > fastest peer first, followed by your next fastest and so on. This works > > best because you get the exponential growth scaling faster by > > propagating the message as "deep" as possible in the network, and it > > then can flood outwards from there. Just sorting the peer list by > > #inv-recevied/time when doing INV pushes and when attending to incoming > > messages would probably be a big improvement. > > In terms of packet size, I would like to look into the network-wide > costs of simply broadcasting block header + coinbase TX + TX list. I > bet miners would love to opt into that. Whether or not that is a improvement is a really complex question, even without taking failure into account. If you agressively prioritize peers that are the most connected and keep your # of peers reasonably low you can afford the memory to keep track of what tx's your peers already know about so to save on round trips for TX hash's they don't have. On the other hand if you have a large number of peers and can't do that, or need to cut down on bandwidth used up by the INV floods and have a probabalistic scheme, you are risking more round-trip latency. Not to mention the nasty problem of how *relying* on TX hashes to keep your bandwidth down means that anything disrupting that system suddenly has a big impact on the network. I don't think we really understand all the nuances of that - look at how few people realize that you need multiples of average bandwidth to have sufficient emergency bandwidth available to catch up in the event of a chain fork. -- 'peter'[:-1]@petertodd.org 00000000000000a1c290ce20953d864a4b9c603abc8a9c77a04429c89c5e9fac [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining 2013-06-10 21:09 ` Peter Todd 2013-06-10 21:23 ` Luke-Jr @ 2013-06-10 21:31 ` Melvin Carvalho 1 sibling, 0 replies; 9+ messages in thread From: Melvin Carvalho @ 2013-06-10 21:31 UTC (permalink / raw) To: Peter Todd; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 3917 bytes --] On 10 June 2013 23:09, Peter Todd <pete@petertodd•org> wrote: > So here's the parts that need to be done for step #1: > > > # Protocol Work > > Basic idea is the miner makes two connections, their pool, and a local > bitcoind. > > They always (usually?) work on the subset of transactions common to both > the pool's getblocktemplate and their local one. When they find a share > if it doesn't meet difficulty they just hand it to the pool. Currently > that is done by handing the whole block over, correct? I know the BIP > says otherwise, but we should optimize this to just hand over tx hashes > where possible. > > If the share does meet difficulty, hand it to both the pool and the > local bitcoind. Should hand it to the pool first though, because the > pool likely has the fastest block propagation, then hand it to local > bitcoind. An optimized version may want to have some record of measured > bandwidth - this applies Bitcoin in general too, although also has other > issues. > > > ## Reducing bandwidth > > How about for normal shares we just pass the block header, and have the > pool randomly pick a subset of transactions to audit? Any fraud cancels > the users shares. This will work best in conjunction with a UTXO proof > tree to prove fees, or by just picking whole shares randomly to audit. > > We'll need persistent share storage so if your connection disconnects > you can provide the pool with the full share later though. > > Incedentally, note how the miner can do the reverse: pick random block > headers and challenge the pool to prove that they correspond to a valid > block. With some modifications Stratum can support this approach. > > > ## Delibrate orphaning of slow to propagate blocks > > Block headers can be flooded-filled much faster than blocks themselves. > They are also small enough to fit into a UDP packet. Nodes should pass > headers around separately via UDP, optinally with some tiny number of > transactions. When we see a valid block header whose contents we do not > know about a miner should switch to mining empty or near empty blocks in > solo mode that would orphan the still propagating block. Doing this is > safe, we'll never build on an invalid block, economically rational while > the inflation subsidy is still high, and helps reduce (although not > eliminate!) the advantage a large miner with high-bandwidth connections > has over those who don't. > > Of course, the other option is to build a block extending the one you > don't know about, which is even more rational, but doing poses major > risks to Bitcoin... > > This functionality can be implemented later - it's not strictly part of > pooled-solo mode. > > > # Pool work > > So does eliopool already accept arbitrary shares like this and do the > correct accounting already? (IE adjust share amount based on fees?) What > happens when the pool doesn't get the share directly, but does see the > new block? > > + possible protocol extensions > > > # Miner work > > Basically we need code to merge the two block templates together to find > commonality. I guess you probably want to implement this in bfgminer > first, so add the code to libblkmaker first, then maybe python-blkmaker. > > We also want an automatic fallback to local solo mining if the pool > can't be contacted. > > + possible protocol extensions > Sounds very promising. Suspect it will need a fair amount of testing ... > > > -- > 'peter'[:-1]@petertodd.org > 000000000000005576673e616271f762a5d8779d5fe7796c6e43ed43df5aa189 > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > [-- Attachment #2: Type: text/html, Size: 5063 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-06-17 17:39 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20130527111149.GB8955@tilt>
[not found] ` <20130531165445.GA29104@petertodd.org>
2013-05-31 16:57 ` [Bitcoin-development] Decentralizing mining Peter Todd
2013-05-31 18:14 ` Adam Back
2013-06-10 21:09 ` Peter Todd
2013-06-10 21:23 ` Luke-Jr
2013-06-14 20:06 ` Peter Todd
2013-06-14 21:05 ` Luke-Jr
2013-06-17 15:16 ` Jeff Garzik
2013-06-17 17:39 ` Peter Todd
2013-06-10 21:31 ` Melvin Carvalho
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox