* [Bitcoin-development] Proposal: SPV Fee Discovery mechanism @ 2015-06-10 17:37 Nathan Wilcox 2015-06-10 19:19 ` Aaron Voisine 2015-06-10 20:26 ` Mike Hearn 0 siblings, 2 replies; 17+ messages in thread From: Nathan Wilcox @ 2015-06-10 17:37 UTC (permalink / raw) To: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 1495 bytes --] [I'm currently wading through bitcoin-development. I'm still about a month behind, so I apologize in advance for any noisy redundancy in this post.] While reading about blocksize, I've just finished Mike Hearn's blog post describing expected systemic behavior as actual blocks approach the current limit (with or without non-protocol-changing implementation improvements): https://medium.com/@octskyward/crash-landing-f5cc19908e32 One detail Mike uses to argue against the "fee's will save us" line of reasoning is that wallets have no good way to learn fee information. So, here's a proposal to fix that: put fee and (and perhaps block size, UTXO, etc...) statistics into the locally-verifiable data available to SPV clients (ie: block headers). It's easy to imagine a hard fork that places details like per-block total fees, transaction count, fee variance, UTXO delta, etc... in a each block header. This would allow SPV clients to rely on this data with the same PoW-backed assurances as all other header data. This mechanism seems valuable regardless of the outcome of blocksize debate. So long as fees are interesting or important, SPV clients should know about them. (Same for other stats such as UTXO count.) Upgrading the protocol without a hard-fork may be possible and is left as an exercise for the reader. ;-) -- Nathan Wilcox Least Authoritarian email: nathan@leastauthority•com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 [-- Attachment #2: Type: text/html, Size: 1898 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 17:37 [Bitcoin-development] Proposal: SPV Fee Discovery mechanism Nathan Wilcox @ 2015-06-10 19:19 ` Aaron Voisine 2015-06-10 20:00 ` Nathan Wilcox 2015-06-10 20:26 ` Mike Hearn 1 sibling, 1 reply; 17+ messages in thread From: Aaron Voisine @ 2015-06-10 19:19 UTC (permalink / raw) To: Nathan Wilcox; +Cc: Bitcoin Development [-- Attachment #1: Type: text/plain, Size: 2821 bytes --] It could be done by agreeing on a data format and encoding it in an op_return output in the coinbase transaction. If it catches on it could later be enforced with a soft fork. For real up-to-the-minute fee calculations you're also going to want to look at the current mempool, how many transactions are waiting, what fees they're paying, etc, but of course that information is susceptible to sybil attack. In practice what we're doing for now is using services like blockcypher who's business is improving reliability of zero-conf to tell us what fee-per-kb is needed, and then putting a hard coded range around it to protect against the service being compromised. This is also the kind of thing being done for exchange rate data which is probably the bigger security risk until bitcoin becomes the standard unit of account for the planet. Aaron Voisine co-founder and CEO breadwallet.com On Wed, Jun 10, 2015 at 10:37 AM, Nathan Wilcox <nathan@leastauthority•com> wrote: > [I'm currently wading through bitcoin-development. I'm still about a month > behind, so I apologize in advance for any noisy redundancy in this post.] > > While reading about blocksize, I've just finished Mike Hearn's blog post > describing expected systemic behavior as actual blocks approach the current > limit (with or without non-protocol-changing implementation improvements): > > https://medium.com/@octskyward/crash-landing-f5cc19908e32 > > > One detail Mike uses to argue against the "fee's will save us" line of > reasoning is that wallets have no good way to learn fee information. > > So, here's a proposal to fix that: put fee and (and perhaps block size, > UTXO, etc...) statistics into the locally-verifiable data available to SPV > clients (ie: block headers). > > > It's easy to imagine a hard fork that places details like per-block total > fees, transaction count, fee variance, UTXO delta, etc... in a each block > header. This would allow SPV clients to rely on this data with the same > PoW-backed assurances as all other header data. > > This mechanism seems valuable regardless of the outcome of blocksize > debate. So long as fees are interesting or important, SPV clients should > know about them. (Same for other stats such as UTXO count.) > > Upgrading the protocol without a hard-fork may be possible and is left as > an exercise for the reader. ;-) > > -- > Nathan Wilcox > Least Authoritarian > > email: nathan@leastauthority•com > twitter: @least_nathan > PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > [-- Attachment #2: Type: text/html, Size: 3987 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 19:19 ` Aaron Voisine @ 2015-06-10 20:00 ` Nathan Wilcox 2015-06-10 20:03 ` Peter Todd 2015-06-10 21:18 ` Aaron Voisine 0 siblings, 2 replies; 17+ messages in thread From: Nathan Wilcox @ 2015-06-10 20:00 UTC (permalink / raw) To: Aaron Voisine; +Cc: Bitcoin Development [-- Attachment #1: Type: text/plain, Size: 4310 bytes --] On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail•com> wrote: > It could be done by agreeing on a data format and encoding it in an > op_return output in the coinbase transaction. If it catches on it could > later be enforced with a soft fork. > > Sounds plausible, except SPV protocols would need to include this coinbase txn if it's going to help SPV clients. (Until a softfork is activated, SPV clients should not rely on this encoding, since until that time the results can be fabricated by individual miners.) > For real up-to-the-minute fee calculations you're also going to want to > look at the current mempool, how many transactions are waiting, what fees > they're paying, etc, but of course that information is susceptible to sybil > attack. > Hm, when you mention Sybil attack, I don't quite follow. When a client relies on any report of a mempool [*], this is already outside the realm of locally-verifiable SPV information, so they are already susceptible to the service making false claims. If that's acceptable (and in many cases it may be) then this whole mechanism is moot, because the client can ask the service for fee statistics for past blocks. > In practice what we're doing for now is using services like blockcypher > who's business is improving reliability of zero-conf to tell us what > fee-per-kb is needed, and then putting a hard coded range around it to > protect against the service being compromised. > This is interesting for me, because I had previously believed fees were fairly static presently, and also because I like hearing about real life wallet implementations. So if this "SPV Fee Stats" feature were added, a wallet might rely on an API for timely stats (aka "block height < 1") then verify that the API isn't lying after doing SPV verification of fee stats for confirmed blocks. This is also the kind of thing being done for exchange rate data which is > probably the bigger security risk until bitcoin becomes the standard unit > of account for the planet. > > That makes sense, although there's no SPV equivalent for exchange data. Aaron Voisine > co-founder and CEO > breadwallet.com > > On Wed, Jun 10, 2015 at 10:37 AM, Nathan Wilcox <nathan@leastauthority•com > > wrote: > >> [I'm currently wading through bitcoin-development. I'm still about a >> month behind, so I apologize in advance for any noisy redundancy in this >> post.] >> >> While reading about blocksize, I've just finished Mike Hearn's blog post >> describing expected systemic behavior as actual blocks approach the current >> limit (with or without non-protocol-changing implementation improvements): >> >> https://medium.com/@octskyward/crash-landing-f5cc19908e32 >> >> >> One detail Mike uses to argue against the "fee's will save us" line of >> reasoning is that wallets have no good way to learn fee information. >> >> So, here's a proposal to fix that: put fee and (and perhaps block size, >> UTXO, etc...) statistics into the locally-verifiable data available to SPV >> clients (ie: block headers). >> >> >> It's easy to imagine a hard fork that places details like per-block total >> fees, transaction count, fee variance, UTXO delta, etc... in a each block >> header. This would allow SPV clients to rely on this data with the same >> PoW-backed assurances as all other header data. >> >> This mechanism seems valuable regardless of the outcome of blocksize >> debate. So long as fees are interesting or important, SPV clients should >> know about them. (Same for other stats such as UTXO count.) >> >> Upgrading the protocol without a hard-fork may be possible and is left as >> an exercise for the reader. ;-) >> >> -- >> Nathan Wilcox >> Least Authoritarian >> >> email: nathan@leastauthority•com >> twitter: @least_nathan >> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 >> >> >> ------------------------------------------------------------------------------ >> >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> >> > -- Nathan Wilcox Least Authoritarian email: nathan@leastauthority•com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 [-- Attachment #2: Type: text/html, Size: 6748 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 20:00 ` Nathan Wilcox @ 2015-06-10 20:03 ` Peter Todd 2015-06-11 18:30 ` Nathan Wilcox 2015-06-10 21:18 ` Aaron Voisine 1 sibling, 1 reply; 17+ messages in thread From: Peter Todd @ 2015-06-10 20:03 UTC (permalink / raw) To: Nathan Wilcox; +Cc: Bitcoin Development [-- Attachment #1: Type: text/plain, Size: 824 bytes --] On Wed, Jun 10, 2015 at 02:00:27PM -0600, Nathan Wilcox wrote: > On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail•com> wrote: > > > It could be done by agreeing on a data format and encoding it in an > > op_return output in the coinbase transaction. If it catches on it could > > later be enforced with a soft fork. > > > > > Sounds plausible, except SPV protocols would need to include this coinbase > txn if it's going to help SPV clients. (Until a softfork is activated, SPV > clients should not rely on this encoding, since until that time the results > can be fabricated by individual miners.) Fee stats can always be fabricated by individual miners because fees can be paid out-of-band. -- 'peter'[:-1]@petertodd.org 00000000000000001245bd2f5c99379ee76836227ded9c08324894faabc0d27f [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 650 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 20:03 ` Peter Todd @ 2015-06-11 18:30 ` Nathan Wilcox 2015-06-11 18:55 ` Aaron Voisine 0 siblings, 1 reply; 17+ messages in thread From: Nathan Wilcox @ 2015-06-11 18:30 UTC (permalink / raw) To: Peter Todd; +Cc: Bitcoin Development [-- Attachment #1: Type: text/plain, Size: 1859 bytes --] On Wed, Jun 10, 2015 at 2:03 PM, Peter Todd <pete@petertodd•org> wrote: > On Wed, Jun 10, 2015 at 02:00:27PM -0600, Nathan Wilcox wrote: > > On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail•com> > wrote: > > > > > It could be done by agreeing on a data format and encoding it in an > > > op_return output in the coinbase transaction. If it catches on it could > > > later be enforced with a soft fork. > > > > > > > > Sounds plausible, except SPV protocols would need to include this > coinbase > > txn if it's going to help SPV clients. (Until a softfork is activated, > SPV > > clients should not rely on this encoding, since until that time the > results > > can be fabricated by individual miners.) > > Fee stats can always be fabricated by individual miners because fees can > be paid out-of-band. > > This is a point I hadn't considered carefully before. I don't understand the marketplace here or why miners would want to move fees outside of explicit inband fees. Implicit in this proposal is that the statistics only cover in-band data, because that's the scope of consensus rules, and thus the proposal is only as useful as the information of in-band fees is useful. I've also noticed a detracting technical argument given a particular tradeoff: A Header-PoW-verifying client could still be given all transactions in a recent block, from which it can see the in-band fees directly. The trade-off is the size of those transactions versus the need to alter any consensus rules or do soft forks. Notice how this trade-off's costs change with maximum block size. > -- > 'peter'[:-1]@petertodd.org > 00000000000000001245bd2f5c99379ee76836227ded9c08324894faabc0d27f > -- Nathan Wilcox Least Authoritarian email: nathan@leastauthority•com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 [-- Attachment #2: Type: text/html, Size: 2839 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-11 18:30 ` Nathan Wilcox @ 2015-06-11 18:55 ` Aaron Voisine 2015-06-13 15:38 ` Nathan Wilcox 0 siblings, 1 reply; 17+ messages in thread From: Aaron Voisine @ 2015-06-11 18:55 UTC (permalink / raw) To: Nathan Wilcox; +Cc: Bitcoin Development [-- Attachment #1: Type: text/plain, Size: 2553 bytes --] > A Header-PoW-verifying client could still be given all transactions in a recent block, from which it can see the in-band fees directly. You don't know the fees paid by any given transaction unless you also have all it's inputs. Transaction inputs do not include an amount. You could however get the average fee-per-kb paid by all transactions in a block by looking at the coinbase transaction, subtracting the block reward, and dividing by the size of block minus the header. Aaron Voisine co-founder and CEO breadwallet.com On Thu, Jun 11, 2015 at 11:30 AM, Nathan Wilcox <nathan@leastauthority•com> wrote: > On Wed, Jun 10, 2015 at 2:03 PM, Peter Todd <pete@petertodd•org> wrote: > >> On Wed, Jun 10, 2015 at 02:00:27PM -0600, Nathan Wilcox wrote: >> > On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail•com> >> wrote: >> > >> > > It could be done by agreeing on a data format and encoding it in an >> > > op_return output in the coinbase transaction. If it catches on it >> could >> > > later be enforced with a soft fork. >> > > >> > > >> > Sounds plausible, except SPV protocols would need to include this >> coinbase >> > txn if it's going to help SPV clients. (Until a softfork is activated, >> SPV >> > clients should not rely on this encoding, since until that time the >> results >> > can be fabricated by individual miners.) >> >> Fee stats can always be fabricated by individual miners because fees can >> be paid out-of-band. >> >> > This is a point I hadn't considered carefully before. I don't understand > the marketplace here or why miners would want to move fees outside of > explicit inband fees. Implicit in this proposal is that the statistics only > cover in-band data, because that's the scope of consensus rules, and thus > the proposal is only as useful as the information of in-band fees is useful. > > I've also noticed a detracting technical argument given a particular > tradeoff: > > A Header-PoW-verifying client could still be given all transactions in a > recent block, from which it can see the in-band fees directly. The > trade-off is the size of those transactions versus the need to alter any > consensus rules or do soft forks. > > Notice how this trade-off's costs change with maximum block size. > > > > >> -- >> 'peter'[:-1]@petertodd.org >> 00000000000000001245bd2f5c99379ee76836227ded9c08324894faabc0d27f >> > > > > -- > Nathan Wilcox > Least Authoritarian > > email: nathan@leastauthority•com > twitter: @least_nathan > PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 > [-- Attachment #2: Type: text/html, Size: 4100 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-11 18:55 ` Aaron Voisine @ 2015-06-13 15:38 ` Nathan Wilcox 0 siblings, 0 replies; 17+ messages in thread From: Nathan Wilcox @ 2015-06-13 15:38 UTC (permalink / raw) To: Aaron Voisine; +Cc: Bitcoin Development [-- Attachment #1: Type: text/plain, Size: 3119 bytes --] On Thu, Jun 11, 2015 at 12:55 PM, Aaron Voisine <voisine@gmail•com> wrote: > > A Header-PoW-verifying client could still be given all transactions in > a recent block, from which it can see the in-band fees directly. > > You don't know the fees paid by any given transaction unless you also have > all it's inputs. Transaction inputs do not include an amount. You could > however get the average fee-per-kb paid by all transactions in a block by > looking at the coinbase transaction, subtracting the block reward, and > dividing by the size of block minus the header. > > Excellent point and alternative proposal. You're right: to get the specifi fees, you'd need all transactions in a block, and all TxOuts with membership proofs. Your alternative seems like a much leaner trade-off for similar data. > > Aaron Voisine > co-founder and CEO > breadwallet.com > > On Thu, Jun 11, 2015 at 11:30 AM, Nathan Wilcox <nathan@leastauthority•com > > wrote: > >> On Wed, Jun 10, 2015 at 2:03 PM, Peter Todd <pete@petertodd•org> wrote: >> >>> On Wed, Jun 10, 2015 at 02:00:27PM -0600, Nathan Wilcox wrote: >>> > On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail•com> >>> wrote: >>> > >>> > > It could be done by agreeing on a data format and encoding it in an >>> > > op_return output in the coinbase transaction. If it catches on it >>> could >>> > > later be enforced with a soft fork. >>> > > >>> > > >>> > Sounds plausible, except SPV protocols would need to include this >>> coinbase >>> > txn if it's going to help SPV clients. (Until a softfork is activated, >>> SPV >>> > clients should not rely on this encoding, since until that time the >>> results >>> > can be fabricated by individual miners.) >>> >>> Fee stats can always be fabricated by individual miners because fees can >>> be paid out-of-band. >>> >>> >> This is a point I hadn't considered carefully before. I don't understand >> the marketplace here or why miners would want to move fees outside of >> explicit inband fees. Implicit in this proposal is that the statistics only >> cover in-band data, because that's the scope of consensus rules, and thus >> the proposal is only as useful as the information of in-band fees is useful. >> >> I've also noticed a detracting technical argument given a particular >> tradeoff: >> >> A Header-PoW-verifying client could still be given all transactions in a >> recent block, from which it can see the in-band fees directly. The >> trade-off is the size of those transactions versus the need to alter any >> consensus rules or do soft forks. >> >> Notice how this trade-off's costs change with maximum block size. >> >> >> >> >>> -- >>> 'peter'[:-1]@petertodd.org >>> 00000000000000001245bd2f5c99379ee76836227ded9c08324894faabc0d27f >>> >> >> >> >> -- >> Nathan Wilcox >> Least Authoritarian >> >> email: nathan@leastauthority•com >> twitter: @least_nathan >> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 >> > > -- Nathan Wilcox Least Authoritarian email: nathan@leastauthority•com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 [-- Attachment #2: Type: text/html, Size: 5236 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 20:00 ` Nathan Wilcox 2015-06-10 20:03 ` Peter Todd @ 2015-06-10 21:18 ` Aaron Voisine 1 sibling, 0 replies; 17+ messages in thread From: Aaron Voisine @ 2015-06-10 21:18 UTC (permalink / raw) To: Nathan Wilcox; +Cc: Bitcoin Development [-- Attachment #1: Type: text/plain, Size: 5306 bytes --] > Sounds plausible, except SPV protocols would need to include this coinbase txn if it's going to help SPV clients. Yes you'd either need a way to add those transactions to the bloom filter, or add/modify a p2p message to request it specifically. > when you mention Sybil attack, I don't quite follow. I just mean that someone could spin up a bunch of malicious p2p nodes that lied about mempool data. It's a bit worse for SPV clients since they can't verify that unconfirmed transactions are valid. > I had previously believed fees were fairly static presently, I actually just added it the other day after getting blockcypher to include it in their api. The current release is still using a hard coded fee rate. Aaron Voisine co-founder and CEO breadwallet.com On Wed, Jun 10, 2015 at 1:00 PM, Nathan Wilcox <nathan@leastauthority•com> wrote: > On Wed, Jun 10, 2015 at 1:19 PM, Aaron Voisine <voisine@gmail•com> wrote: > >> It could be done by agreeing on a data format and encoding it in an >> op_return output in the coinbase transaction. If it catches on it could >> later be enforced with a soft fork. >> >> > Sounds plausible, except SPV protocols would need to include this coinbase > txn if it's going to help SPV clients. (Until a softfork is activated, SPV > clients should not rely on this encoding, since until that time the results > can be fabricated by individual miners.) > > >> For real up-to-the-minute fee calculations you're also going to want to >> look at the current mempool, how many transactions are waiting, what fees >> they're paying, etc, but of course that information is susceptible to sybil >> attack. >> > > Hm, when you mention Sybil attack, I don't quite follow. > > When a client relies on any report of a mempool [*], this is already > outside the realm of locally-verifiable SPV information, so they are > already susceptible to the service making false claims. If that's > acceptable (and in many cases it may be) then this whole mechanism is moot, > because the client can ask the service for fee statistics for past blocks. > > >> In practice what we're doing for now is using services like blockcypher >> who's business is improving reliability of zero-conf to tell us what >> fee-per-kb is needed, and then putting a hard coded range around it to >> protect against the service being compromised. >> > > This is interesting for me, because I had previously believed fees were > fairly static presently, and also because I like hearing about real life > wallet implementations. > > So if this "SPV Fee Stats" feature were added, a wallet might rely on an > API for timely stats (aka "block height < 1") then verify that the API > isn't lying after doing SPV verification of fee stats for confirmed blocks. > > > This is also the kind of thing being done for exchange rate data which is >> probably the bigger security risk until bitcoin becomes the standard unit >> of account for the planet. >> >> > That makes sense, although there's no SPV equivalent for exchange data. > > > Aaron Voisine >> co-founder and CEO >> breadwallet.com >> >> On Wed, Jun 10, 2015 at 10:37 AM, Nathan Wilcox < >> nathan@leastauthority•com> wrote: >> >>> [I'm currently wading through bitcoin-development. I'm still about a >>> month behind, so I apologize in advance for any noisy redundancy in this >>> post.] >>> >>> While reading about blocksize, I've just finished Mike Hearn's blog post >>> describing expected systemic behavior as actual blocks approach the current >>> limit (with or without non-protocol-changing implementation improvements): >>> >>> https://medium.com/@octskyward/crash-landing-f5cc19908e32 >>> >>> >>> One detail Mike uses to argue against the "fee's will save us" line of >>> reasoning is that wallets have no good way to learn fee information. >>> >>> So, here's a proposal to fix that: put fee and (and perhaps block size, >>> UTXO, etc...) statistics into the locally-verifiable data available to SPV >>> clients (ie: block headers). >>> >>> >>> It's easy to imagine a hard fork that places details like per-block >>> total fees, transaction count, fee variance, UTXO delta, etc... in a each >>> block header. This would allow SPV clients to rely on this data with the >>> same PoW-backed assurances as all other header data. >>> >>> This mechanism seems valuable regardless of the outcome of blocksize >>> debate. So long as fees are interesting or important, SPV clients should >>> know about them. (Same for other stats such as UTXO count.) >>> >>> Upgrading the protocol without a hard-fork may be possible and is left >>> as an exercise for the reader. ;-) >>> >>> -- >>> Nathan Wilcox >>> Least Authoritarian >>> >>> email: nathan@leastauthority•com >>> twitter: @least_nathan >>> PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Bitcoin-development mailing list >>> Bitcoin-development@lists•sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>> >>> >> > > > -- > Nathan Wilcox > Least Authoritarian > > email: nathan@leastauthority•com > twitter: @least_nathan > PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 > [-- Attachment #2: Type: text/html, Size: 8651 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 17:37 [Bitcoin-development] Proposal: SPV Fee Discovery mechanism Nathan Wilcox 2015-06-10 19:19 ` Aaron Voisine @ 2015-06-10 20:26 ` Mike Hearn 2015-06-10 21:18 ` Aaron Voisine 1 sibling, 1 reply; 17+ messages in thread From: Mike Hearn @ 2015-06-10 20:26 UTC (permalink / raw) To: Nathan Wilcox; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 1076 bytes --] I described an alternative way for SPV wallets to learn about fees some time ago. It requires a new transaction version that embeds output values into the signed data. Then an upgrade to the P2P protocol to send UTXO data along with transactions when they are relayed. The idea is that the wallet sets a Bloom filter with an FP rate that ensures it will see some random subset of all transactions being broadcast on the network, and with the extra data, it can calculate the fee paid. Once a transaction broadcast is observed the wallet includes that tx hash in its next Bloom filter, thus it can see which block the tx confirmed in. By measuring the amount of time that passed between a broadcast and it appearing in a block, it can calculate its own tables of fee paid:time taken. This has the advantage that you don't have to trust miners to publish data accurately. However it requires some protocol upgrades and of course, a lot of new code in SPV wallets. The way Bitcoin Wallet for Android handles fees currently is to just update a hard coded value every so often. [-- Attachment #2: Type: text/html, Size: 1174 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 20:26 ` Mike Hearn @ 2015-06-10 21:18 ` Aaron Voisine 2015-06-11 10:19 ` Mike Hearn 2015-06-11 13:10 ` Peter Todd 0 siblings, 2 replies; 17+ messages in thread From: Aaron Voisine @ 2015-06-10 21:18 UTC (permalink / raw) To: Mike Hearn; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 1932 bytes --] The other complication is that this will tend to be a lagging indicator based on network congestion from the last time you connected. If we assume that transactions are being dropped in an unpredictable way when blocks are full, knowing the network congestion *right now* is critical, and even then you just have to hope that someone who wants that space more than you do doesn't show up after you disconnect. Aaron Voisine co-founder and CEO breadwallet.com On Wed, Jun 10, 2015 at 1:26 PM, Mike Hearn <mike@plan99•net> wrote: > I described an alternative way for SPV wallets to learn about fees some > time ago. It requires a new transaction version that embeds output values > into the signed data. Then an upgrade to the P2P protocol to send UTXO data > along with transactions when they are relayed. > > The idea is that the wallet sets a Bloom filter with an FP rate that > ensures it will see some random subset of all transactions being broadcast > on the network, and with the extra data, it can calculate the fee paid. > Once a transaction broadcast is observed the wallet includes that tx hash > in its next Bloom filter, thus it can see which block the tx confirmed in. > By measuring the amount of time that passed between a broadcast and it > appearing in a block, it can calculate its own tables of fee paid:time > taken. > > This has the advantage that you don't have to trust miners to publish data > accurately. However it requires some protocol upgrades and of course, a lot > of new code in SPV wallets. > > The way Bitcoin Wallet for Android handles fees currently is to just > update a hard coded value every so often. > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > [-- Attachment #2: Type: text/html, Size: 2657 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 21:18 ` Aaron Voisine @ 2015-06-11 10:19 ` Mike Hearn 2015-06-11 13:10 ` Peter Todd 1 sibling, 0 replies; 17+ messages in thread From: Mike Hearn @ 2015-06-11 10:19 UTC (permalink / raw) To: Aaron Voisine; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 500 bytes --] > > If we assume that transactions are being dropped in an unpredictable way > when blocks are full, knowing the network congestion *right now* is > critical, and even then you just have to hope that someone who wants that > space more than you do doesn't show up after you disconnect. > Yeah, my proposal is not intended to function correctly with full blocks, as Bitcoin cannot work at all in such a state. It assumes that fees only change slowly and that transactions are being cleared normally. [-- Attachment #2: Type: text/html, Size: 736 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-10 21:18 ` Aaron Voisine 2015-06-11 10:19 ` Mike Hearn @ 2015-06-11 13:10 ` Peter Todd 2015-06-11 14:11 ` Martin Lie ` (2 more replies) 1 sibling, 3 replies; 17+ messages in thread From: Peter Todd @ 2015-06-11 13:10 UTC (permalink / raw) To: Aaron Voisine; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 850 bytes --] On Wed, Jun 10, 2015 at 02:18:30PM -0700, Aaron Voisine wrote: > The other complication is that this will tend to be a lagging indicator > based on network congestion from the last time you connected. If we assume > that transactions are being dropped in an unpredictable way when blocks are > full, knowing the network congestion *right now* is critical, and even then > you just have to hope that someone who wants that space more than you do > doesn't show up after you disconnect. Hence the need for ways to increase fees on transactions after initial broadcast like replace-by-fee and child-pays-for-parent. Re: "dropped in an unpredictable way" - transactions would be dropped lowest fee/KB first, a completely predictable way. -- 'peter'[:-1]@petertodd.org 0000000000000000124bae79afdcee9267b4e6f8137758b8b4135455cd8e3bfd [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 650 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-11 13:10 ` Peter Todd @ 2015-06-11 14:11 ` Martin Lie 2015-06-11 17:10 ` Tom Harding 2015-06-11 18:18 ` Nathan Wilcox 2 siblings, 0 replies; 17+ messages in thread From: Martin Lie @ 2015-06-11 14:11 UTC (permalink / raw) To: Bitcoin Dev Peter Todd wrote: > Re: "dropped in an unpredictable way" - transactions would be dropped > lowest fee/KB first, a completely predictable way. It would be 'completely predictable' for whoever knew the state and policies of a miner's mempool, but from an end user's perspective that wouldn't matter much: The end users wouldn't know if their transaction(s) would make the cut or not, somewhere in the network, and by what time. They (obviously) won't know what miners will find the next block(s), they won't know the miners' mempool sizes, potential custom eviction policies, etc. I agree that this can be somewhat remedied by FSSRBF/CPFP, though, provided wallets give users a good (semi-automated?) interface for such transaction replacements/chains. -- Martin Lie ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-11 13:10 ` Peter Todd 2015-06-11 14:11 ` Martin Lie @ 2015-06-11 17:10 ` Tom Harding 2015-06-11 17:52 ` Mike Hearn 2015-06-11 18:18 ` Nathan Wilcox 2 siblings, 1 reply; 17+ messages in thread From: Tom Harding @ 2015-06-11 17:10 UTC (permalink / raw) To: bitcoin-development On 6/11/2015 6:10 AM, Peter Todd wrote: > On Wed, Jun 10, 2015 at 02:18:30PM -0700, Aaron Voisine wrote: >> The other complication is that this will tend to be a lagging indicator >> based on network congestion from the last time you connected. If we assume >> that transactions are being dropped in an unpredictable way when blocks are >> full, knowing the network congestion *right now* is critical, and even then >> you just have to hope that someone who wants that space more than you do >> doesn't show up after you disconnect. > Hence the need for ways to increase fees on transactions after initial > broadcast like replace-by-fee and child-pays-for-parent. > > Re: "dropped in an unpredictable way" - transactions would be dropped > lowest fee/KB first, a completely predictable way. Quite agreed. Also, transactions with unconfirmed inputs should be among the first to get dropped, as discussed in the "Dropped-transaction spam" thread. Like all policy rules, either of these works in proportion to its deployment. Be advised that pull request #6068 emphasizes the view that the network will never have consistent mempool/relay policies, and on the contrary needs a framework that supports and encourages pluggable, generally parameterized policies that could (some might say should) conflict wildly with each other. It probably doesn't matter that much. Deploying a new policy still wouldn't be much easier than deploying a patched version. I mean, nobody has proposed a policy rule engine yet (oops). ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-11 17:10 ` Tom Harding @ 2015-06-11 17:52 ` Mike Hearn 2015-06-12 6:44 ` Aaron Voisine 0 siblings, 1 reply; 17+ messages in thread From: Mike Hearn @ 2015-06-11 17:52 UTC (permalink / raw) To: Tom Harding; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 315 bytes --] > > > Re: "dropped in an unpredictable way" - transactions would be dropped > > lowest fee/KB first, a completely predictable way. > > Quite agreed. No, Aaron is correct. It's unpredictable from the perspective of the user sending the transaction, and as they are the ones picking the fees, that is what matters. [-- Attachment #2: Type: text/html, Size: 588 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-11 17:52 ` Mike Hearn @ 2015-06-12 6:44 ` Aaron Voisine 0 siblings, 0 replies; 17+ messages in thread From: Aaron Voisine @ 2015-06-12 6:44 UTC (permalink / raw) To: Mike Hearn; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 1209 bytes --] One possible solution that wallets could adopt when blocks fill up, would be to abandon the p2p network for transaction propagation altogether, and instead work directly with a handful of the largest miners and pools to get transactions into blocks. The miners could auction off space in their blocks with the guarantee that they will be included in the order accepted. We'd lose the peer-to-peer nature of sending transactions for a federated service operator style model, but avoid the problem of unpredictable transaction failure. Also, unlike replace-by-fee, this would allow for a send-and-forget usage pattern, as well as known up-front fees. Something like this is certainly what I imagine the large hosted wallet companies will end up moving to. Aaron On Thursday, June 11, 2015, Mike Hearn <mike@plan99•net> wrote: > > Re: "dropped in an unpredictable way" - transactions would be dropped >> > lowest fee/KB first, a completely predictable way. >> >> Quite agreed. > > > No, Aaron is correct. It's unpredictable from the perspective of the user > sending the transaction, and as they are the ones picking the fees, that is > what matters. > -- Aaron Voisine co-founder and CEO breadwallet.com [-- Attachment #2: Type: text/html, Size: 1834 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Bitcoin-development] Proposal: SPV Fee Discovery mechanism 2015-06-11 13:10 ` Peter Todd 2015-06-11 14:11 ` Martin Lie 2015-06-11 17:10 ` Tom Harding @ 2015-06-11 18:18 ` Nathan Wilcox 2 siblings, 0 replies; 17+ messages in thread From: Nathan Wilcox @ 2015-06-11 18:18 UTC (permalink / raw) To: Peter Todd; +Cc: Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 2812 bytes --] On Thu, Jun 11, 2015 at 7:10 AM, Peter Todd <pete@petertodd•org> wrote: > On Wed, Jun 10, 2015 at 02:18:30PM -0700, Aaron Voisine wrote: > > The other complication is that this will tend to be a lagging indicator > > based on network congestion from the last time you connected. If we > assume > > that transactions are being dropped in an unpredictable way when blocks > are > > full, knowing the network congestion *right now* is critical, and even > then > > you just have to hope that someone who wants that space more than you do > > doesn't show up after you disconnect. > > Hence the need for ways to increase fees on transactions after initial > broadcast like replace-by-fee and child-pays-for-parent. > > I haven't looked closely at replace-by-fee yet, but I assume this is a non-consensus change to mempool mechanics. To me, this seems like the "actuator" side of fee mechanics: it provides a transaction sender a way to influence the system. By contrast, learning about fees is the "sensor" side of fee mechanics. Consider how a replace-by-fee wallet makes fee decisions. When does it replace by fee? It needs feedback in one of two forms: a. Direct feedback from a trnasaction relay service, or: b. Information in the blockchain, which is verified by all verifying nodes and refined by all miners. The first kind of information is quite acceptable and practical for many use cases, but leave the wallet vulnerable to fabrications by that service. This vulnerability is precisely what SPV security intends to mitigate, right? With only information type a, a (non-SPV) wallet can "shop around" to find competing services, and this should work pretty well, provided the wallet can discover those competing services. If, OTOH, the wallet has access to information type b, it now has "perfect competition" across all such services, even when it can't discover the low-priced services directly. This actual-fees-in-the-actual-block-chain information of type b seem like a powerful source of pricing information, and if SPV security is already valuable for other reasons, then it seems natural to leverage that value. Re: "dropped in an unpredictable way" - transactions would be dropped > lowest fee/KB first, a completely predictable way. > > -- > 'peter'[:-1]@petertodd.org > 0000000000000000124bae79afdcee9267b4e6f8137758b8b4135455cd8e3bfd > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > -- Nathan Wilcox Least Authoritarian email: nathan@leastauthority•com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993 [-- Attachment #2: Type: text/html, Size: 3956 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2015-06-13 15:39 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-06-10 17:37 [Bitcoin-development] Proposal: SPV Fee Discovery mechanism Nathan Wilcox 2015-06-10 19:19 ` Aaron Voisine 2015-06-10 20:00 ` Nathan Wilcox 2015-06-10 20:03 ` Peter Todd 2015-06-11 18:30 ` Nathan Wilcox 2015-06-11 18:55 ` Aaron Voisine 2015-06-13 15:38 ` Nathan Wilcox 2015-06-10 21:18 ` Aaron Voisine 2015-06-10 20:26 ` Mike Hearn 2015-06-10 21:18 ` Aaron Voisine 2015-06-11 10:19 ` Mike Hearn 2015-06-11 13:10 ` Peter Todd 2015-06-11 14:11 ` Martin Lie 2015-06-11 17:10 ` Tom Harding 2015-06-11 17:52 ` Mike Hearn 2015-06-12 6:44 ` Aaron Voisine 2015-06-11 18:18 ` Nathan Wilcox
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox