* [Bitcoin-development] Decentralizing mining
[not found] ` <20130531165445.GA29104@petertodd.org>
@ 2013-05-31 16:57 ` Peter Todd
2013-05-31 18:14 ` Adam Back
2013-06-10 21:09 ` Peter Todd
0 siblings, 2 replies; 9+ messages in thread
From: Peter Todd @ 2013-05-31 16:57 UTC (permalink / raw)
To: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 2841 bytes --]
I just posted the following to bitcointalk.
https://bitcointalk.org/index.php?topic=221164.0
Right now between two to four running the largest pools control Bitcoin
in the short term. That's a lot of hashing power in the hands of very,
very few people. In addition pools have little incentive to run secure
operations, and many pools have been hacked with their funds stolen.
Those hacks could just have easily been used to attack the network
itself.
This needs to change.
Pooled-solo mining is a concept Gregory Maxwell, Luke Dashjr and I were
discussing at the conference two weeks ago. (credit goes to Greg and
Luke; I was mostly just listening) Basically the idea is that miners
with mining equipment run a local Bitcoin node and use that node to
construct the blocks they mine - the same as if they were solo mining.
The pools job is then to only track shares and organize payouts.
If the pool gets hacked the worst that can happen is miners are ripped
off, rather than Bitcoin itself being attacked. With pooled-solo mining
even a pool with a majority of hashing power wouldn't be able to do much
harm to Bitcoin. (depending on the implementation they may be able to
blacklist specific transactions - the pool needs to know what
transactions are in the share to credit fees properly)
Tech-wise Luke created getblocktemplate last year as a means to audit
mining pools. I'm sure Greg and Luke can explain the nitty gritty
details better than I can, but essentially the plan is to take
getblocktemplate and extend it as required for pooled-solo mining. This
will include pool and miner software initially, and later improvements
to GUIs and what not to make the whole process easier.
With the success of my recent video project I also want to make this
Keep Bitcoin Free's next project, specifically funding a developer
(likely Luke) to make this happen. Additionally once software is written
and easily usable a good follow-up would be a video and other media to
promote the idea to miners. No guarantees we'll be able to come up with
commercially competitive remuneration, but we can at least come up with
a "Thank you" tip. But first lets discuss the technical requirements to
get an idea of what the scope is.
Finally, for the record, a big part of the reason why myself and other
Keep Bitcoin Free supporters are interested in doing this is very much
to take power over the direction of the network from big pools and put
it into the hands of thousands of individual miners. It's much easier to
convince people that changes to Bitcoin, like increasing the blocksize,
are directly impacting decentralization when individual miners are
seeing that happen to themselves.
--
'peter'[:-1]@petertodd.org
00000000000000c14fa7031b2431ab32785efdf1e5aaecc83555ee52a2fc550b
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining
2013-05-31 16:57 ` [Bitcoin-development] Decentralizing mining Peter Todd
@ 2013-05-31 18:14 ` Adam Back
2013-06-10 21:09 ` Peter Todd
1 sibling, 0 replies; 9+ messages in thread
From: Adam Back @ 2013-05-31 18:14 UTC (permalink / raw)
To: Peter Todd; +Cc: bitcoin-development
I like this idea a lot.
To add: I think it is a bug and security risk if pooled-solo or (current
pooled miners) do not add randomness to their extraNonce2 (like 128-bits of
it). For privacy and to avoid various hostile-pre-mining attacks it should
be done this way. Lack of the self-chosen challenge field is the reason
Satoshi's first year mining is marked (plus forgetting to reset the
counter). (Bitcoind I believe considered the direct miners key as defense
enough as a stand in for self-chosen challenge, which has a few problems).
The base counter I think is only 32-bits, the extranonce2 itself being
random can be incremented while still looking random. But incrementing
extranonce directy while initializing it to 0 is not good (per previous
mining extranone marked coins bug - is that even fixed?)
(You dont want to reveal the miners power in his pool shares, if the full
counter is revealed with no randomness it also reveals how many iterations
he can do since the block start).
Adam
On Fri, May 31, 2013 at 12:57:58PM -0400, Peter Todd wrote:
>I just posted the following to bitcointalk.
>
>https://bitcointalk.org/index.php?topic=221164.0
>
>
>Right now between two to four running the largest pools control Bitcoin
>in the short term. That's a lot of hashing power in the hands of very,
>very few people. In addition pools have little incentive to run secure
>operations, and many pools have been hacked with their funds stolen.
>Those hacks could just have easily been used to attack the network
>itself.
>
>This needs to change.
>
>Pooled-solo mining is a concept Gregory Maxwell, Luke Dashjr and I were
>discussing at the conference two weeks ago. (credit goes to Greg and
>Luke; I was mostly just listening) Basically the idea is that miners
>with mining equipment run a local Bitcoin node and use that node to
>construct the blocks they mine - the same as if they were solo mining.
>The pools job is then to only track shares and organize payouts.
>
>If the pool gets hacked the worst that can happen is miners are ripped
>off, rather than Bitcoin itself being attacked. With pooled-solo mining
>even a pool with a majority of hashing power wouldn't be able to do much
>harm to Bitcoin. (depending on the implementation they may be able to
>blacklist specific transactions - the pool needs to know what
>transactions are in the share to credit fees properly)
>
>Tech-wise Luke created getblocktemplate last year as a means to audit
>mining pools. I'm sure Greg and Luke can explain the nitty gritty
>details better than I can, but essentially the plan is to take
>getblocktemplate and extend it as required for pooled-solo mining. This
>will include pool and miner software initially, and later improvements
>to GUIs and what not to make the whole process easier.
>
>
>With the success of my recent video project I also want to make this
>Keep Bitcoin Free's next project, specifically funding a developer
>(likely Luke) to make this happen. Additionally once software is written
>and easily usable a good follow-up would be a video and other media to
>promote the idea to miners. No guarantees we'll be able to come up with
>commercially competitive remuneration, but we can at least come up with
>a "Thank you" tip. But first lets discuss the technical requirements to
>get an idea of what the scope is.
>
>
>Finally, for the record, a big part of the reason why myself and other
>Keep Bitcoin Free supporters are interested in doing this is very much
>to take power over the direction of the network from big pools and put
>it into the hands of thousands of individual miners. It's much easier to
>convince people that changes to Bitcoin, like increasing the blocksize,
>are directly impacting decentralization when individual miners are
>seeing that happen to themselves.
>
>--
>'peter'[:-1]@petertodd.org
>00000000000000c14fa7031b2431ab32785efdf1e5aaecc83555ee52a2fc550b
>------------------------------------------------------------------------------
>Get 100% visibility into Java/.NET code with AppDynamics Lite
>It's a free troubleshooting tool designed for production
>Get down to code-level detail for bottlenecks, with <2% overhead.
>Download for free and get started troubleshooting in minutes.
>http://p.sf.net/sfu/appdyn_d2d_ap2
>_______________________________________________
>Bitcoin-development mailing list
>Bitcoin-development@lists•sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/bitcoin-development
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining
2013-05-31 16:57 ` [Bitcoin-development] Decentralizing mining Peter Todd
2013-05-31 18:14 ` Adam Back
@ 2013-06-10 21:09 ` Peter Todd
2013-06-10 21:23 ` Luke-Jr
2013-06-10 21:31 ` Melvin Carvalho
1 sibling, 2 replies; 9+ messages in thread
From: Peter Todd @ 2013-06-10 21:09 UTC (permalink / raw)
To: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 3319 bytes --]
So here's the parts that need to be done for step #1:
# Protocol Work
Basic idea is the miner makes two connections, their pool, and a local
bitcoind.
They always (usually?) work on the subset of transactions common to both
the pool's getblocktemplate and their local one. When they find a share
if it doesn't meet difficulty they just hand it to the pool. Currently
that is done by handing the whole block over, correct? I know the BIP
says otherwise, but we should optimize this to just hand over tx hashes
where possible.
If the share does meet difficulty, hand it to both the pool and the
local bitcoind. Should hand it to the pool first though, because the
pool likely has the fastest block propagation, then hand it to local
bitcoind. An optimized version may want to have some record of measured
bandwidth - this applies Bitcoin in general too, although also has other
issues.
## Reducing bandwidth
How about for normal shares we just pass the block header, and have the
pool randomly pick a subset of transactions to audit? Any fraud cancels
the users shares. This will work best in conjunction with a UTXO proof
tree to prove fees, or by just picking whole shares randomly to audit.
We'll need persistent share storage so if your connection disconnects
you can provide the pool with the full share later though.
Incedentally, note how the miner can do the reverse: pick random block
headers and challenge the pool to prove that they correspond to a valid
block. With some modifications Stratum can support this approach.
## Delibrate orphaning of slow to propagate blocks
Block headers can be flooded-filled much faster than blocks themselves.
They are also small enough to fit into a UDP packet. Nodes should pass
headers around separately via UDP, optinally with some tiny number of
transactions. When we see a valid block header whose contents we do not
know about a miner should switch to mining empty or near empty blocks in
solo mode that would orphan the still propagating block. Doing this is
safe, we'll never build on an invalid block, economically rational while
the inflation subsidy is still high, and helps reduce (although not
eliminate!) the advantage a large miner with high-bandwidth connections
has over those who don't.
Of course, the other option is to build a block extending the one you
don't know about, which is even more rational, but doing poses major
risks to Bitcoin...
This functionality can be implemented later - it's not strictly part of
pooled-solo mode.
# Pool work
So does eliopool already accept arbitrary shares like this and do the
correct accounting already? (IE adjust share amount based on fees?) What
happens when the pool doesn't get the share directly, but does see the
new block?
+ possible protocol extensions
# Miner work
Basically we need code to merge the two block templates together to find
commonality. I guess you probably want to implement this in bfgminer
first, so add the code to libblkmaker first, then maybe python-blkmaker.
We also want an automatic fallback to local solo mining if the pool
can't be contacted.
+ possible protocol extensions
--
'peter'[:-1]@petertodd.org
000000000000005576673e616271f762a5d8779d5fe7796c6e43ed43df5aa189
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining
2013-06-10 21:09 ` Peter Todd
@ 2013-06-10 21:23 ` Luke-Jr
2013-06-14 20:06 ` Peter Todd
2013-06-10 21:31 ` Melvin Carvalho
1 sibling, 1 reply; 9+ messages in thread
From: Luke-Jr @ 2013-06-10 21:23 UTC (permalink / raw)
To: bitcoin-development
On Monday, June 10, 2013 9:09:13 PM Peter Todd wrote:
> # Protocol Work
This is basically done.
> Basic idea is the miner makes two connections, their pool, and a local
> bitcoind.
>
> They always (usually?) work on the subset of transactions common to both
> the pool's getblocktemplate and their local one. When they find a share
> if it doesn't meet difficulty they just hand it to the pool. Currently
> that is done by handing the whole block over, correct? I know the BIP
> says otherwise, but we should optimize this to just hand over tx hashes
> where possible.
The plan was to tell the pool it doesn't need to send transactions at all, and
only work on the ones from bitcoind. Currently, share submissions are just the
block header and coinbase transaction; in this case, however, the miner will
need to send merkle links also, possibly just once via a block proposal in
advance.
> If the share does meet difficulty, hand it to both the pool and the
> local bitcoind. Should hand it to the pool first though, because the
> pool likely has the fastest block propagation, then hand it to local
> bitcoind. An optimized version may want to have some record of measured
> bandwidth - this applies Bitcoin in general too, although also has other
> issues.
Currently, BFGMiner is doing submission to the pool, waiting for a response,
then submitting to a local bitcoind. This is because the pool might need to
receive/record the share before it processes the block on bitcoind, or you
could lose credit for it. The response from the pool is rather small (a single
TCP packet), so this shouldn't delay much longer.
> ## Reducing bandwidth
>
> How about for normal shares we just pass the block header, and have the
> pool randomly pick a subset of transactions to audit? Any fraud cancels
> the users shares. This will work best in conjunction with a UTXO proof
> tree to prove fees, or by just picking whole shares randomly to audit.
Might as well just use higher difficulty shares (each one audited) for the
same effect. Block proposals allow the miner to tell the pool its transaction
set once (per txset change) for any number of shares.
IF bandwidth becomes a real problem, I have a draft of a "GBT 2.0" that does
some more improvement in this area.
> # Pool work
>
> So does eliopool already accept arbitrary shares like this and do the
> correct accounting already? (IE adjust share amount based on fees?) What
> happens when the pool doesn't get the share directly, but does see the
> new block?
>
> + possible protocol extensions
I don't follow.
> # Miner work
>
> Basically we need code to merge the two block templates together to find
> commonality. I guess you probably want to implement this in bfgminer
> first, so add the code to libblkmaker first, then maybe python-blkmaker.
libblkmaker's API was designed for this from the start, so it should be fairly
easily implemented.
> We also want an automatic fallback to local solo mining if the pool
> can't be contacted.
>
> + possible protocol extensions
Failover already functions, but probably could use a rewrite...
Luke
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining
2013-06-10 21:09 ` Peter Todd
2013-06-10 21:23 ` Luke-Jr
@ 2013-06-10 21:31 ` Melvin Carvalho
1 sibling, 0 replies; 9+ messages in thread
From: Melvin Carvalho @ 2013-06-10 21:31 UTC (permalink / raw)
To: Peter Todd; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 3917 bytes --]
On 10 June 2013 23:09, Peter Todd <pete@petertodd•org> wrote:
> So here's the parts that need to be done for step #1:
>
>
> # Protocol Work
>
> Basic idea is the miner makes two connections, their pool, and a local
> bitcoind.
>
> They always (usually?) work on the subset of transactions common to both
> the pool's getblocktemplate and their local one. When they find a share
> if it doesn't meet difficulty they just hand it to the pool. Currently
> that is done by handing the whole block over, correct? I know the BIP
> says otherwise, but we should optimize this to just hand over tx hashes
> where possible.
>
> If the share does meet difficulty, hand it to both the pool and the
> local bitcoind. Should hand it to the pool first though, because the
> pool likely has the fastest block propagation, then hand it to local
> bitcoind. An optimized version may want to have some record of measured
> bandwidth - this applies Bitcoin in general too, although also has other
> issues.
>
>
> ## Reducing bandwidth
>
> How about for normal shares we just pass the block header, and have the
> pool randomly pick a subset of transactions to audit? Any fraud cancels
> the users shares. This will work best in conjunction with a UTXO proof
> tree to prove fees, or by just picking whole shares randomly to audit.
>
> We'll need persistent share storage so if your connection disconnects
> you can provide the pool with the full share later though.
>
> Incedentally, note how the miner can do the reverse: pick random block
> headers and challenge the pool to prove that they correspond to a valid
> block. With some modifications Stratum can support this approach.
>
>
> ## Delibrate orphaning of slow to propagate blocks
>
> Block headers can be flooded-filled much faster than blocks themselves.
> They are also small enough to fit into a UDP packet. Nodes should pass
> headers around separately via UDP, optinally with some tiny number of
> transactions. When we see a valid block header whose contents we do not
> know about a miner should switch to mining empty or near empty blocks in
> solo mode that would orphan the still propagating block. Doing this is
> safe, we'll never build on an invalid block, economically rational while
> the inflation subsidy is still high, and helps reduce (although not
> eliminate!) the advantage a large miner with high-bandwidth connections
> has over those who don't.
>
> Of course, the other option is to build a block extending the one you
> don't know about, which is even more rational, but doing poses major
> risks to Bitcoin...
>
> This functionality can be implemented later - it's not strictly part of
> pooled-solo mode.
>
>
> # Pool work
>
> So does eliopool already accept arbitrary shares like this and do the
> correct accounting already? (IE adjust share amount based on fees?) What
> happens when the pool doesn't get the share directly, but does see the
> new block?
>
> + possible protocol extensions
>
>
> # Miner work
>
> Basically we need code to merge the two block templates together to find
> commonality. I guess you probably want to implement this in bfgminer
> first, so add the code to libblkmaker first, then maybe python-blkmaker.
>
> We also want an automatic fallback to local solo mining if the pool
> can't be contacted.
>
> + possible protocol extensions
>
Sounds very promising. Suspect it will need a fair amount of testing ...
>
>
> --
> 'peter'[:-1]@petertodd.org
> 000000000000005576673e616271f762a5d8779d5fe7796c6e43ed43df5aa189
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
[-- Attachment #2: Type: text/html, Size: 5063 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining
2013-06-10 21:23 ` Luke-Jr
@ 2013-06-14 20:06 ` Peter Todd
2013-06-14 21:05 ` Luke-Jr
2013-06-17 15:16 ` Jeff Garzik
0 siblings, 2 replies; 9+ messages in thread
From: Peter Todd @ 2013-06-14 20:06 UTC (permalink / raw)
To: Luke-Jr; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 5609 bytes --]
On Mon, Jun 10, 2013 at 09:23:14PM +0000, Luke-Jr wrote:
> > They always (usually?) work on the subset of transactions common to both
> > the pool's getblocktemplate and their local one. When they find a share
> > if it doesn't meet difficulty they just hand it to the pool. Currently
> > that is done by handing the whole block over, correct? I know the BIP
> > says otherwise, but we should optimize this to just hand over tx hashes
> > where possible.
>
> The plan was to tell the pool it doesn't need to send transactions at all, and
> only work on the ones from bitcoind. Currently, share submissions are just the
> block header and coinbase transaction; in this case, however, the miner will
> need to send merkle links also, possibly just once via a block proposal in
> advance.
It strikes me that this would work best if the pool has a mempool with
child-pays-for-parent support where conflicts *are* allowed.
IE you record whatever transactions you know about, conflicting or not,
calculate which ones gives you the most fees/kb, and then figure out
which set of non-conflicting ones are optimal. Of course, "optimal" is
the knapsack problem...
Now you can easilly tell the miners working on shares for you which tx's
would be optimal if they wish to know, and at the same time whatever
shares they send you are most likely to include transactions in your
mempool inventory, and thus they can send just tx hashes to reduce
bandwidth.
Part of the broader issue that when we send peers INV advertisements we
should be telling them what the fee/kb is so our peers can prioritize
properly. That'd also help for the case where you want to broadcast two
transactions in a row, but the pair is only profitable because the
second is paying the fee for the first.
Speaking of, the way we tell peers about new blocks is really
suboptimal: we tell every peer, in no particular order, about a new
block via a block INV message, and then we give them the new block in
parallel. I was looking through comp-sci papers on optimal
flood-fill/gossip algorithms for random graph networks and it appears
that optimal is to spend all your bandwidth to send the message to your
fastest peer first, followed by your next fastest and so on. This works
best because you get the exponential growth scaling faster by
propagating the message as "deep" as possible in the network, and it
then can flood outwards from there. Just sorting the peer list by
#inv-recevied/time when doing INV pushes and when attending to incoming
messages would probably be a big improvement.
> > If the share does meet difficulty, hand it to both the pool and the
> > local bitcoind. Should hand it to the pool first though, because the
> > pool likely has the fastest block propagation, then hand it to local
> > bitcoind. An optimized version may want to have some record of measured
> > bandwidth - this applies Bitcoin in general too, although also has other
> > issues.
>
> Currently, BFGMiner is doing submission to the pool, waiting for a response,
> then submitting to a local bitcoind. This is because the pool might need to
> receive/record the share before it processes the block on bitcoind, or you
> could lose credit for it. The response from the pool is rather small (a single
> TCP packet), so this shouldn't delay much longer.
Right, I guess the pool wants to be sure you were actually the one who
found the share, rather than just someone who was lucky enough to see it
on the network and submitted it as your own.
> > ## Reducing bandwidth
> >
> > How about for normal shares we just pass the block header, and have the
> > pool randomly pick a subset of transactions to audit? Any fraud cancels
> > the users shares. This will work best in conjunction with a UTXO proof
> > tree to prove fees, or by just picking whole shares randomly to audit.
>
> Might as well just use higher difficulty shares (each one audited) for the
> same effect. Block proposals allow the miner to tell the pool its transaction
> set once (per txset change) for any number of shares.
That's a good point - the current practice most pools seem to follow of
about a share per second seems very excessive to me. On the other hand,
it does have good user optics. The right solution might be something
akin to P2Pool where the UI level is telling the user shares are being
found so it's clear "stuff is happening", but under the hood only a
small subset are ever sent to the pool.
> > # Pool work
> >
> > So does eliopool already accept arbitrary shares like this and do the
> > correct accounting already? (IE adjust share amount based on fees?) What
> > happens when the pool doesn't get the share directly, but does see the
> > new block?
> >
> > + possible protocol extensions
>
> I don't follow.
What part don't you follow?
> > # Miner work
> >
> > Basically we need code to merge the two block templates together to find
> > commonality. I guess you probably want to implement this in bfgminer
> > first, so add the code to libblkmaker first, then maybe python-blkmaker.
>
> libblkmaker's API was designed for this from the start, so it should be fairly
> easily implemented.
Good.
> > We also want an automatic fallback to local solo mining if the pool
> > can't be contacted.
> >
> > + possible protocol extensions
>
> Failover already functions, but probably could use a rewrite...
Sounds good.
--
'peter'[:-1]@petertodd.org
000000000000006ed83d97e7266a66dcb057cd53750795bd209076fa0a34d4ec
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining
2013-06-14 20:06 ` Peter Todd
@ 2013-06-14 21:05 ` Luke-Jr
2013-06-17 15:16 ` Jeff Garzik
1 sibling, 0 replies; 9+ messages in thread
From: Luke-Jr @ 2013-06-14 21:05 UTC (permalink / raw)
To: Peter Todd; +Cc: bitcoin-development
On Friday, June 14, 2013 8:06:54 PM Peter Todd wrote:
> On Mon, Jun 10, 2013 at 09:23:14PM +0000, Luke-Jr wrote:
> > Might as well just use higher difficulty shares (each one audited) for
> > the same effect. Block proposals allow the miner to tell the pool its
> > transaction set once (per txset change) for any number of shares.
>
> That's a good point - the current practice most pools seem to follow of
> about a share per second seems very excessive to me. On the other hand,
> it does have good user optics. The right solution might be something
> akin to P2Pool where the UI level is telling the user shares are being
> found so it's clear "stuff is happening", but under the hood only a
> small subset are ever sent to the pool.
Share rate is relevant to more than user information - it also affects the
variance of reward/payout. I disagree with claiming shares are found when
they're not sent to the pool - this makes auditing and troubleshooting more
difficult; for example, see the GUIMiner bug where it reports shares despite
misinterpreting the pool's target and submitting nothing at all (this happens
when the pool uses pdiff 1).
> > > # Pool work
> > >
> > > So does eliopool already accept arbitrary shares like this and do the
> > > correct accounting already? (IE adjust share amount based on fees?)
> > > What happens when the pool doesn't get the share directly, but does
> > > see the new block?
> > >
> > > + possible protocol extensions
> >
> > I don't follow.
>
> What part don't you follow?
I don't understand the first two questions here at all.
Luke
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining
2013-06-14 20:06 ` Peter Todd
2013-06-14 21:05 ` Luke-Jr
@ 2013-06-17 15:16 ` Jeff Garzik
2013-06-17 17:39 ` Peter Todd
1 sibling, 1 reply; 9+ messages in thread
From: Jeff Garzik @ 2013-06-17 15:16 UTC (permalink / raw)
To: Peter Todd; +Cc: bitcoin-development
On Fri, Jun 14, 2013 at 4:06 PM, Peter Todd <pete@petertodd•org> wrote:
> It strikes me that this would work best if the pool has a mempool with
> child-pays-for-parent support where conflicts *are* allowed.
>
> IE you record whatever transactions you know about, conflicting or not,
> calculate which ones gives you the most fees/kb, and then figure out
> which set of non-conflicting ones are optimal. Of course, "optimal" is
> the knapsack problem...
>
> Now you can easilly tell the miners working on shares for you which tx's
> would be optimal if they wish to know, and at the same time whatever
> shares they send you are most likely to include transactions in your
> mempool inventory, and thus they can send just tx hashes to reduce
> bandwidth.
>
>
> Part of the broader issue that when we send peers INV advertisements we
> should be telling them what the fee/kb is so our peers can prioritize
> properly. That'd also help for the case where you want to broadcast two
> transactions in a row, but the pair is only profitable because the
> second is paying the fee for the first.
Interesting proposals, particularly this last. The net result impact
is, however, that which was criticized in at least one forum thread:
replace-with-higher-fee.
> Speaking of, the way we tell peers about new blocks is really
> suboptimal: we tell every peer, in no particular order, about a new
> block via a block INV message, and then we give them the new block in
> parallel. I was looking through comp-sci papers on optimal
> flood-fill/gossip algorithms for random graph networks and it appears
> that optimal is to spend all your bandwidth to send the message to your
> fastest peer first, followed by your next fastest and so on. This works
> best because you get the exponential growth scaling faster by
> propagating the message as "deep" as possible in the network, and it
> then can flood outwards from there. Just sorting the peer list by
> #inv-recevied/time when doing INV pushes and when attending to incoming
> messages would probably be a big improvement.
In terms of packet size, I would like to look into the network-wide
costs of simply broadcasting block header + coinbase TX + TX list. I
bet miners would love to opt into that.
>> > If the share does meet difficulty, hand it to both the pool and the
>> > local bitcoind. Should hand it to the pool first though, because the
>> > pool likely has the fastest block propagation, then hand it to local
>> > bitcoind. An optimized version may want to have some record of measured
>> > bandwidth - this applies Bitcoin in general too, although also has other
>> > issues.
>>
>> Currently, BFGMiner is doing submission to the pool, waiting for a response,
>> then submitting to a local bitcoind. This is because the pool might need to
>> receive/record the share before it processes the block on bitcoind, or you
>> could lose credit for it. The response from the pool is rather small (a single
>> TCP packet), so this shouldn't delay much longer.
>
> Right, I guess the pool wants to be sure you were actually the one who
> found the share, rather than just someone who was lucky enough to see it
> on the network and submitted it as your own.
>
>> > ## Reducing bandwidth
>> >
>> > How about for normal shares we just pass the block header, and have the
>> > pool randomly pick a subset of transactions to audit? Any fraud cancels
>> > the users shares. This will work best in conjunction with a UTXO proof
>> > tree to prove fees, or by just picking whole shares randomly to audit.
>>
>> Might as well just use higher difficulty shares (each one audited) for the
>> same effect. Block proposals allow the miner to tell the pool its transaction
>> set once (per txset change) for any number of shares.
>
> That's a good point - the current practice most pools seem to follow of
> about a share per second seems very excessive to me. On the other hand,
> it does have good user optics. The right solution might be something
> akin to P2Pool where the UI level is telling the user shares are being
> found so it's clear "stuff is happening", but under the hood only a
> small subset are ever sent to the pool.
With the onslaught of ASIC mining, most big pools are past a share per
second. Variable difficulty or set-to-higher-difficulty quickly
became the norm, almost out of necessity.
Personally, I think most pools should target at _least_ 5-10 seconds
per share, no matter the strength of the miner.
--
Jeff Garzik
Senior Software Engineer and open source evangelist
BitPay, Inc. https://bitpay.com/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bitcoin-development] Decentralizing mining
2013-06-17 15:16 ` Jeff Garzik
@ 2013-06-17 17:39 ` Peter Todd
0 siblings, 0 replies; 9+ messages in thread
From: Peter Todd @ 2013-06-17 17:39 UTC (permalink / raw)
To: Jeff Garzik; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 3564 bytes --]
On Mon, Jun 17, 2013 at 11:16:01AM -0400, Jeff Garzik wrote:
> > Part of the broader issue that when we send peers INV advertisements we
> > should be telling them what the fee/kb is so our peers can prioritize
> > properly. That'd also help for the case where you want to broadcast two
> > transactions in a row, but the pair is only profitable because the
> > second is paying the fee for the first.
>
> Interesting proposals, particularly this last. The net result impact
> is, however, that which was criticized in at least one forum thread:
> replace-with-higher-fee.
Actually the two are orthogonal: a low-priority no-fee tx might result
because it was from a customer paying a merchant via the payment
protocol. The merchant can then respend that tx with a fee to cover
both, but with the current mempool arrangement if the no-fee tx load is
high actually getting that first tx to propagate so the second can will
be difficult.
A nice way to do this would be to accept tx's into your mempool
indiscriminately but delay broadcasting INV messages until you find
child tx's that make the low-profit ones worth mining. When you do find
a child with a sufficiently high fee, send an INVGROUP message to notify
your peers of the new opportunity. Different nodes will have different
ideas of what priority TX deserves to be broadcast, but here provided
the group meets the threshold a peer will always find out.
> > Speaking of, the way we tell peers about new blocks is really
> > suboptimal: we tell every peer, in no particular order, about a new
> > block via a block INV message, and then we give them the new block in
> > parallel. I was looking through comp-sci papers on optimal
> > flood-fill/gossip algorithms for random graph networks and it appears
> > that optimal is to spend all your bandwidth to send the message to your
> > fastest peer first, followed by your next fastest and so on. This works
> > best because you get the exponential growth scaling faster by
> > propagating the message as "deep" as possible in the network, and it
> > then can flood outwards from there. Just sorting the peer list by
> > #inv-recevied/time when doing INV pushes and when attending to incoming
> > messages would probably be a big improvement.
>
> In terms of packet size, I would like to look into the network-wide
> costs of simply broadcasting block header + coinbase TX + TX list. I
> bet miners would love to opt into that.
Whether or not that is a improvement is a really complex question, even
without taking failure into account. If you agressively prioritize peers
that are the most connected and keep your # of peers reasonably low you
can afford the memory to keep track of what tx's your peers already know
about so to save on round trips for TX hash's they don't have. On the
other hand if you have a large number of peers and can't do that, or
need to cut down on bandwidth used up by the INV floods and have a
probabalistic scheme, you are risking more round-trip latency.
Not to mention the nasty problem of how *relying* on TX hashes to keep
your bandwidth down means that anything disrupting that system suddenly
has a big impact on the network. I don't think we really understand all
the nuances of that - look at how few people realize that you need
multiples of average bandwidth to have sufficient emergency bandwidth
available to catch up in the event of a chain fork.
--
'peter'[:-1]@petertodd.org
00000000000000a1c290ce20953d864a4b9c603abc8a9c77a04429c89c5e9fac
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-06-17 17:39 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20130527111149.GB8955@tilt>
[not found] ` <20130531165445.GA29104@petertodd.org>
2013-05-31 16:57 ` [Bitcoin-development] Decentralizing mining Peter Todd
2013-05-31 18:14 ` Adam Back
2013-06-10 21:09 ` Peter Todd
2013-06-10 21:23 ` Luke-Jr
2013-06-14 20:06 ` Peter Todd
2013-06-14 21:05 ` Luke-Jr
2013-06-17 15:16 ` Jeff Garzik
2013-06-17 17:39 ` Peter Todd
2013-06-10 21:31 ` Melvin Carvalho
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox