* [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
@ 2016-05-09 8:26 bfd
2016-05-09 8:57 ` Gregory Maxwell
` (3 more replies)
0 siblings, 4 replies; 34+ messages in thread
From: bfd @ 2016-05-09 8:26 UTC (permalink / raw)
To: bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
Thea properties of BIP37 SPV [0] are unfortunately not as strong as
originally thought:
* The expected privacy of the probabilistic nature of bloom
filters does not exist [1][2], any user with a BIP37 SPV wallet
should be operating under no expectation of privacy.
Implementation flaws make this effect significantly worse, the
behavior meaning that no matter how high the false positive
rate (up to simply downloading the whole blocks verbatim) the
intent of the client connection is recoverable.
* Significant processing load is placed on nodes in the Bitcoin
network by lightweight clients, a single syncing wallet causes
(at the time of writing) 80GB of disk reads and a large amount
of CPU time to be consumed processing this data. This carries
significant denial of service risk [3], non-distinguishable
clients can repeatedly request taxing blocks causing
reprocessing on every request. Processed data is unique to every
client, and can not be cached or made more efficient while
staying within specification.
* Wallet clients can not have strong consistency or security
expectations, BIP37 merkle paths allow for a wallet to validate
that an output was spendable at some point in time but does not
prove that this output is not spent today.
* Nodes in the network can denial of service attack all BIP37 SPV
wallet clients by simply returning null filter results for
requests, the wallet has no way of discerning if it has been
lied to and may be made simply unaware that any payment has been
made to them. Many nodes can be queried in a probabilistic manor
but this increases the already heavy network load with little
benefit.
We propose a new concept which can work towards addressing these
shortcomings.
A Bloom Filter Digest is deterministically created of every block
encompassing the inputs and outputs of the containing transactions,
the filter parameters being tuned such that the filter is a small
portion of the size of the total block data. To determine if a block
has contents which may be interesting a second bloom filter of all
relevant key material is created. A binary comparison between the two
filters returns true if there is probably matching transactions, and
false if there is certainly no matching transactions. Any matched
blocks can be downloaded in full and processed for transactions which
may be relevant.
The BFD can be used verbatim in replacement of BIP37, where the filter
can be cached between clients without needing to be recomputed. It can
also be used by normal pruned nodes to do re-scans locally of their
wallet without needing to have the block data available to scan, or
without reading the entire block chain from disk.
-
For improved probabilistic security the bloom filters can be presented
to lightweight clients by semi-trusted oracles. A client wallet makes
an assumption that they trust a set, or subset of remote parties
(wallet vendors, services) which all all sign the BFD for each block.
The BFD can be downloaded from a single remote source, and the hash of
the filters compared against others in the trust set. Agreement is a
weak suggestion that the filter has not been tampered with, assuming
that these parties are not conspiring to defraud the client.
The oracles do not learn any additional information about the client
wallet, the client can download the block data from either nodes on
the network, HTTP services, NTTP, or any other out of band
communication method that provides the privacy desired by the client.
-
The security model of the oracle bloom filter can be vastly improved
by instead committing a hash of the BFD inside every block as a soft-
fork consensus rule change. After this, every node in the network would
build the filter and validate that the hash in the block is correct,
then make a conscious choice discard it for space savings or cache the
data to disk.
With a commitment to the filter it becomes impossible to lie to
lightweight clients by omission. Lightweight clients are provided with
a block header, merkle path, and the BFD. Altering the BFD invalidates
the merkle proof, it's validity is a strong indicator that the client
has an unadulterated picture of the UTXO condition without needing to
build one itself. A strong assurance that the hash of the BFD means
that the filters can be downloaded out of band along with the block
data at the leisure of the client, allowing for significantly greater
privacy and taking load away from the P2P Bitcoin network.
Committing the BFD is not a hard forking change, and does not require
alterations to mining software so long as the coinbase transaction
scriptSig is not included in the bloom filter.
[0] https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki
[1] https://eprint.iacr.org/2014/763.pdf
[2] https://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj/
[3] https://github.com/petertodd/bloom-io-attack
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2016-05-09 8:26 [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security bfd
@ 2016-05-09 8:57 ` Gregory Maxwell
2016-05-11 20:06 ` Bob McElrath
` (2 subsequent siblings)
3 siblings, 0 replies; 34+ messages in thread
From: Gregory Maxwell @ 2016-05-09 8:57 UTC (permalink / raw)
To: bfd; +Cc: Bitcoin Dev
On Mon, May 9, 2016 at 8:26 AM, bfd--- via bitcoin-dev
<bitcoin-dev@lists•linuxfoundation.org> wrote:
> We introduce several concepts that rework the lightweight Bitcoin
> client model in a manner which is secure, efficient and privacy
> compatible.
[...]
> A Bloom Filter Digest is deterministically created of every block
I think this is a fantastic idea.
Some napkin work shows that it has pretty good communications
bandwidth so long as you assume that the wallet has many keys (e.g.
more than the number of the outputs in the block)-- otherwise BIP37
uses less bandwidth, but you note its terrible privacy problems.
You should be aware that when the filter is transmitted but not
updated, as it is in these filtering applications, the bloom filter is
not the most communication efficient data structure.
The most efficient data structure is similar to a bloom filter, but
you use more bits and only one hash function. The result will be
mostly zero bits. Then you entropy code it using RLE+Rice coding or an
optimal binomial packer (e.g.
https://people.xiph.org/~greg/binomial_codec.c). This is about 45%
more space efficient than a bloom filter. ... it's just a PITA to
update, though that is inapplicable here. Entropy coding for this can
be quite fast, if many lookups are done the decompression could even
be faster than having to use two dozen hash functions for each lookup.
The intuition is that this kind of simple hash-bitmap is great, but
space inefficient if you don't have compression since most of the bits
are 0 you end up spending a bit to send less than a bit of
information. A bloom filter improve the situation by using the
multiple filters to increase the ones density to 50%, but the
increased collisions create overhead. This is important when its a
in-memory data-structure that you're updating often, but not here.
One thing to do with matching blocks is after finding the matches the
node could potentially consult some PIR to get the blocks it cares
about... thus preventing a leak of which blocks it was interested in,
but not taking PIR costs for the whole chain or requiring the
implementation of PIR tree search (which is theoretically simple but
in practice hard to implement).
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2016-05-09 8:26 [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security bfd
2016-05-09 8:57 ` Gregory Maxwell
@ 2016-05-11 20:06 ` Bob McElrath
2016-05-11 20:29 ` Bob McElrath
[not found] ` <77b6dd25-0603-a0bd-6a9e-38098e5cb19d@jonasschnelli.ch>
2017-02-17 0:28 ` Chris Belcher
3 siblings, 1 reply; 34+ messages in thread
From: Bob McElrath @ 2016-05-11 20:06 UTC (permalink / raw)
To: bfd; +Cc: bitcoin-dev
[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]
I like this idea, but let's run some numbers...
bfd--- via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org] wrote:
> A Bloom Filter Digest is deterministically created of every block
Bloom filters completely obfuscate the required size of the filter for a desired
false-positive rate. But, an optimal filter is linear in the number of elements
it contains for fixed false-positive rate, and logarithmic in the false-positive
rate. (This comment applies to a RLL encoded Bloom filter Greg mentioned, but
that's not the only way) That is for N elements and false positive rate
\epsilon:
filter size = - N \log_2 \epsilon
Given that the data that would be put into this particular filter is *already*
hashed, it makes more sense and is faster to use a Cuckoo[1] filter, choosing a
fixed false-positive rate, given expected wallet sizes. For Bloom filters,
multiply the above formula by 1.44.
To prevent light clients from downloading more blocks than necessary, the
false-positive rate should be roughly less than 1/(block height). If we take
the false positive rate to be 1e-6 for today's block height ~ 410000, this is
about 20 bits per element. So for todays block's, this is a 30kb filter, for a
3% increase in block size, if blocks commit to the filter. Thus the required
size of the filter commitment is roughly:
filter size = N \log_2 H
where H is the block height. If bitcoin had these filters from the beginning, a
light client today would have to download about 12MB of data in filters. My
personal SPV wallet is using 31MB currently. It's not clear this is a bandwidth
win, though it's definitely a win for computing load on full nodes.
[1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2016-05-11 20:06 ` Bob McElrath
@ 2016-05-11 20:29 ` Bob McElrath
2016-07-28 21:07 ` Leo Wandersleb
2017-01-03 20:24 ` bfd
0 siblings, 2 replies; 34+ messages in thread
From: Bob McElrath @ 2016-05-11 20:29 UTC (permalink / raw)
To: Bob McElrath, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 2711 bytes --]
Eerrrr....let me revise that last paragraph. That's 12 *GB* of filters at
today's block height (at fixed false-positive rate 1e-6. Compared to block
headers only which are about 33 MB today. So this proposal is not really
compatible with such a wallet being "light"...
Damn units...
Bob McElrath via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org] wrote:
> I like this idea, but let's run some numbers...
>
> bfd--- via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org] wrote:
> > A Bloom Filter Digest is deterministically created of every block
>
> Bloom filters completely obfuscate the required size of the filter for a desired
> false-positive rate. But, an optimal filter is linear in the number of elements
> it contains for fixed false-positive rate, and logarithmic in the false-positive
> rate. (This comment applies to a RLL encoded Bloom filter Greg mentioned, but
> that's not the only way) That is for N elements and false positive rate
> \epsilon:
>
> filter size = - N \log_2 \epsilon
>
> Given that the data that would be put into this particular filter is *already*
> hashed, it makes more sense and is faster to use a Cuckoo[1] filter, choosing a
> fixed false-positive rate, given expected wallet sizes. For Bloom filters,
> multiply the above formula by 1.44.
>
> To prevent light clients from downloading more blocks than necessary, the
> false-positive rate should be roughly less than 1/(block height). If we take
> the false positive rate to be 1e-6 for today's block height ~ 410000, this is
> about 20 bits per element. So for todays block's, this is a 30kb filter, for a
> 3% increase in block size, if blocks commit to the filter. Thus the required
> size of the filter commitment is roughly:
>
> filter size = N \log_2 H
>
> where H is the block height. If bitcoin had these filters from the beginning, a
> light client today would have to download about 12MB of data in filters. My
> personal SPV wallet is using 31MB currently. It's not clear this is a bandwidth
> win, though it's definitely a win for computing load on full nodes.
>
>
> [1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
>
> --
> Cheers, Bob McElrath
>
> "For every complex problem, there is a solution that is simple, neat, and wrong."
> -- H. L. Mencken
>
>
>
> !DSPAM:5733934b206851108912031!
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
> !DSPAM:5733934b206851108912031!
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2016-05-11 20:29 ` Bob McElrath
@ 2016-07-28 21:07 ` Leo Wandersleb
2017-01-06 22:07 ` Erik Aronesty
2017-01-03 20:24 ` bfd
1 sibling, 1 reply; 34+ messages in thread
From: Leo Wandersleb @ 2016-07-28 21:07 UTC (permalink / raw)
To: bitcoin-dev
[-- Attachment #1: Type: text/plain, Size: 4016 bytes --]
gmaxwell just made me aware of this mail thread [0]. Some days ago I had
independently and naively started implementing "something similar" [1].
My version totally ignored the commitment and signing part but I'm pretty sure
that 12GB is overkill. My code is currently broken and I have no time to work on
it much but I thought it might be helpful to chime in.
At this point in time the difference between 80GB and 3GB (as my current 1.5GB
of only outputs would suggest if I had covered the inputs) or even 12GB makes
the difference of being able to store it on a phone, vs. not being able to. 80GB
"compressed" to 3GB is not that bad at all. Unfortunately, with segWit this will
be worse, with the higher transaction count per MB.
Regards,
Leo
[0]
https://www.reddit.com/r/Bitcoin/comments/4v28jl/how_have_fungiblity_problems_affected_you_in/d5ux6aq
[1] https://github.com/Giszmo/TransactionFinder
On 05/11/2016 10:29 PM, Bob McElrath via bitcoin-dev wrote:
> Eerrrr....let me revise that last paragraph. That's 12 *GB* of filters at
> today's block height (at fixed false-positive rate 1e-6. Compared to block
> headers only which are about 33 MB today. So this proposal is not really
> compatible with such a wallet being "light"...
>
> Damn units...
>
> Bob McElrath via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org] wrote:
>> I like this idea, but let's run some numbers...
>>
>> bfd--- via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org] wrote:
>>> A Bloom Filter Digest is deterministically created of every block
>> Bloom filters completely obfuscate the required size of the filter for a desired
>> false-positive rate. But, an optimal filter is linear in the number of elements
>> it contains for fixed false-positive rate, and logarithmic in the false-positive
>> rate. (This comment applies to a RLL encoded Bloom filter Greg mentioned, but
>> that's not the only way) That is for N elements and false positive rate
>> \epsilon:
>>
>> filter size = - N \log_2 \epsilon
>>
>> Given that the data that would be put into this particular filter is *already*
>> hashed, it makes more sense and is faster to use a Cuckoo[1] filter, choosing a
>> fixed false-positive rate, given expected wallet sizes. For Bloom filters,
>> multiply the above formula by 1.44.
>>
>> To prevent light clients from downloading more blocks than necessary, the
>> false-positive rate should be roughly less than 1/(block height). If we take
>> the false positive rate to be 1e-6 for today's block height ~ 410000, this is
>> about 20 bits per element. So for todays block's, this is a 30kb filter, for a
>> 3% increase in block size, if blocks commit to the filter. Thus the required
>> size of the filter commitment is roughly:
>>
>> filter size = N \log_2 H
>>
>> where H is the block height. If bitcoin had these filters from the beginning, a
>> light client today would have to download about 12MB of data in filters. My
>> personal SPV wallet is using 31MB currently. It's not clear this is a bandwidth
>> win, though it's definitely a win for computing load on full nodes.
>>
>>
>> [1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
>>
>> --
>> Cheers, Bob McElrath
>>
>> "For every complex problem, there is a solution that is simple, neat, and wrong."
>> -- H. L. Mencken
>>
>>
>>
>> !DSPAM:5733934b206851108912031!
>
>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>> !DSPAM:5733934b206851108912031!
> --
> Cheers, Bob McElrath
>
> "For every complex problem, there is a solution that is simple, neat, and wrong."
> -- H. L. Mencken
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
[not found] ` <77b6dd25-0603-a0bd-6a9e-38098e5cb19d@jonasschnelli.ch>
@ 2017-01-03 20:18 ` bfd
2017-01-03 22:18 ` Aaron Voisine
0 siblings, 1 reply; 34+ messages in thread
From: bfd @ 2017-01-03 20:18 UTC (permalink / raw)
To: Jonas Schnelli, Bitcoin Protocol Discussion
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
On 2017-01-01 13:01, Jonas Schnelli via bitcoin-dev wrote:
> Hi
>> We introduce several concepts that rework the lightweight Bitcoin
>> client model in a manner which is secure, efficient and privacy
>> compatible.
>>
>> The BFD can be used verbatim in replacement of BIP37, where the filter
>> can be cached between clients without needing to be recomputed. It can
>> also be used by normal pruned nodes to do re-scans locally of their
>> wallet without needing to have the block data available to scan, or
>> without reading the entire block chain from disk.
> I started exploring the potential of BFD after this specification.
>
> What would be the preferred/recommended way to handle 0-conf/mempool
> filtering – if & once BDF would have been deployed (any type,
> semi-trusted oracles or protocol-level/softfork)?
>
> From the user-experience perspective, this is probably pretty important
> (otherwise the experience will be that incoming funds can take serval
> minutes to hours until they appear).
> Using BIP37 bloom filters just for mempool filtering would obviously
> result in the same unwanted privacy-setup.
>
> </jonas>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2016-05-11 20:29 ` Bob McElrath
2016-07-28 21:07 ` Leo Wandersleb
@ 2017-01-03 20:24 ` bfd
1 sibling, 0 replies; 34+ messages in thread
From: bfd @ 2017-01-03 20:24 UTC (permalink / raw)
To: Bob McElrath; +Cc: Bitcoin Protocol Discussion
I believe the filter can be more compact than this, but even if not an
order of magnitude saving of disk space is still significant.
On 2016-05-11 13:29, Bob McElrath wrote:
> Eerrrr....let me revise that last paragraph. That's 12 *GB* of filters
> at
> today's block height (at fixed false-positive rate 1e-6. Compared to
> block
> headers only which are about 33 MB today. So this proposal is not
> really
> compatible with such a wallet being "light"...
>
> Damn units...
>
> Bob McElrath via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org]
> wrote:
>> I like this idea, but let's run some numbers...
>>
>> bfd--- via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org] wrote:
>> > A Bloom Filter Digest is deterministically created of every block
>>
>> Bloom filters completely obfuscate the required size of the filter for
>> a desired
>> false-positive rate. But, an optimal filter is linear in the number
>> of elements
>> it contains for fixed false-positive rate, and logarithmic in the
>> false-positive
>> rate. (This comment applies to a RLL encoded Bloom filter Greg
>> mentioned, but
>> that's not the only way) That is for N elements and false positive
>> rate
>> \epsilon:
>>
>> filter size = - N \log_2 \epsilon
>>
>> Given that the data that would be put into this particular filter is
>> *already*
>> hashed, it makes more sense and is faster to use a Cuckoo[1] filter,
>> choosing a
>> fixed false-positive rate, given expected wallet sizes. For Bloom
>> filters,
>> multiply the above formula by 1.44.
>>
>> To prevent light clients from downloading more blocks than necessary,
>> the
>> false-positive rate should be roughly less than 1/(block height). If
>> we take
>> the false positive rate to be 1e-6 for today's block height ~ 410000,
>> this is
>> about 20 bits per element. So for todays block's, this is a 30kb
>> filter, for a
>> 3% increase in block size, if blocks commit to the filter. Thus the
>> required
>> size of the filter commitment is roughly:
>>
>> filter size = N \log_2 H
>>
>> where H is the block height. If bitcoin had these filters from the
>> beginning, a
>> light client today would have to download about 12MB of data in
>> filters. My
>> personal SPV wallet is using 31MB currently. It's not clear this is a
>> bandwidth
>> win, though it's definitely a win for computing load on full nodes.
>>
>>
>> [1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
>>
>> --
>> Cheers, Bob McElrath
>>
>> "For every complex problem, there is a solution that is simple, neat,
>> and wrong."
>> -- H. L. Mencken
>>
>>
>>
>> !DSPAM:5733934b206851108912031!
>
>
>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>> !DSPAM:5733934b206851108912031!
>
> --
> Cheers, Bob McElrath
>
> "For every complex problem, there is a solution that is simple, neat,
> and wrong."
> -- H. L. Mencken
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-03 20:18 ` bfd
@ 2017-01-03 22:18 ` Aaron Voisine
2017-01-03 22:28 ` bfd
` (2 more replies)
0 siblings, 3 replies; 34+ messages in thread
From: Aaron Voisine @ 2017-01-03 22:18 UTC (permalink / raw)
To: Bitcoin Protocol Discussion, Jonas Schnelli, bfd
[-- Attachment #1: Type: text/plain, Size: 2769 bytes --]
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large cost
to losing the ability to have instant transactions in many or even most
situations. This cost is typically well above the fraud risk.
It's important to recognize that bitcoin serves a wide variety of use cases
with different profiles for time sensitivity and fraud risk.
Aaron
On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:
> The concept combined with the weak blocks system where miners commit
>
> to potential transaction inclusion with fractional difficulty blocks
>
> is possible. I'm not personally convinced that unconfirmed transaction
>
> display in a wallet is worth the privacy trade-off. The user has very
>
> little to gain from this knowledge until the txn is in a block.
>
>
>
>
>
> On 2017-01-01 13:01, Jonas Schnelli via bitcoin-dev wrote:
>
> > Hi
>
> >> We introduce several concepts that rework the lightweight Bitcoin
>
> >> client model in a manner which is secure, efficient and privacy
>
> >> compatible.
>
> >>
>
> >> The BFD can be used verbatim in replacement of BIP37, where the filter
>
> >> can be cached between clients without needing to be recomputed. It can
>
> >> also be used by normal pruned nodes to do re-scans locally of their
>
> >> wallet without needing to have the block data available to scan, or
>
> >> without reading the entire block chain from disk.
>
> > I started exploring the potential of BFD after this specification.
>
> >
>
> > What would be the preferred/recommended way to handle 0-conf/mempool
>
> > filtering – if & once BDF would have been deployed (any type,
>
> > semi-trusted oracles or protocol-level/softfork)?
>
> >
>
> > From the user-experience perspective, this is probably pretty important
>
> > (otherwise the experience will be that incoming funds can take serval
>
> > minutes to hours until they appear).
>
> > Using BIP37 bloom filters just for mempool filtering would obviously
>
> > result in the same unwanted privacy-setup.
>
> >
>
> > </jonas>
>
> >
>
> >
>
> > _______________________________________________
>
> > bitcoin-dev mailing list
>
> > bitcoin-dev@lists•linuxfoundation.org
>
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
> _______________________________________________
>
> bitcoin-dev mailing list
>
> bitcoin-dev@lists•linuxfoundation.org
>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
[-- Attachment #2: Type: text/html, Size: 4405 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-03 22:18 ` Aaron Voisine
@ 2017-01-03 22:28 ` bfd
2017-01-03 23:06 ` adiabat
2017-01-04 7:47 ` Jonas Schnelli
2 siblings, 0 replies; 34+ messages in thread
From: bfd @ 2017-01-03 22:28 UTC (permalink / raw)
To: Aaron Voisine; +Cc: Bitcoin Protocol Discussion
The concept was not particularly targeted towards businesses, but
allowing for significantly improved wallet performance and reducing
privacy for lite clients. You would expect that a business has the
capacity to run a fully validating, fully storing node of their own.
If they’re not something is fundamentally broken with Bitcoin, or
their rationale of continuing to use it.
On 2017-01-03 14:18, Aaron Voisine wrote:
> Unconfirmed transactions are incredibly important for real world use.
> Merchants for instance are willing to accept credit card payments of
> thousands of dollars and ship the goods despite the fact that the
> transaction can be reversed up to 60 days later. There is a very large
> cost to losing the ability to have instant transactions in many or
> even most situations. This cost is typically well above the fraud
> risk.
>
> It's important to recognize that bitcoin serves a wide variety of use
> cases with different profiles for time sensitivity and fraud risk.
>
> Aaron
>
> On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev
> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>
>> The concept combined with the weak blocks system where miners commit
>>
>> to potential transaction inclusion with fractional difficulty blocks
>>
>> is possible. I'm not personally convinced that unconfirmed
>> transaction
>>
>> display in a wallet is worth the privacy trade-off. The user has
>> very
>>
>> little to gain from this knowledge until the txn is in a block.
>>
>> On 2017-01-01 13:01, Jonas Schnelli via bitcoin-dev wrote:
>>
>>> Hi
>>
>>>> We introduce several concepts that rework the lightweight Bitcoin
>>
>>>> client model in a manner which is secure, efficient and privacy
>>
>>>> compatible.
>>
>>>>
>>
>>>> The BFD can be used verbatim in replacement of BIP37, where the
>> filter
>>
>>>> can be cached between clients without needing to be recomputed.
>> It can
>>
>>>> also be used by normal pruned nodes to do re-scans locally of
>> their
>>
>>>> wallet without needing to have the block data available to scan,
>> or
>>
>>>> without reading the entire block chain from disk.
>>
>>> I started exploring the potential of BFD after this specification.
>>
>>>
>>
>>> What would be the preferred/recommended way to handle
>> 0-conf/mempool
>>
>>> filtering – if & once BDF would have been deployed (any type,
>>
>>> semi-trusted oracles or protocol-level/softfork)?
>>
>>>
>>
>>> From the user-experience perspective, this is probably pretty
>> important
>>
>>> (otherwise the experience will be that incoming funds can take
>> serval
>>
>>> minutes to hours until they appear).
>>
>>> Using BIP37 bloom filters just for mempool filtering would
>> obviously
>>
>>> result in the same unwanted privacy-setup.
>>
>>>
>>
>>> </jonas>
>>
>>>
>>
>>>
>>
>>> _______________________________________________
>>
>>> bitcoin-dev mailing list
>>
>>> bitcoin-dev@lists•linuxfoundation.org
>>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> _______________________________________________
>>
>> bitcoin-dev mailing list
>>
>> bitcoin-dev@lists•linuxfoundation.org
>>
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-03 22:18 ` Aaron Voisine
2017-01-03 22:28 ` bfd
@ 2017-01-03 23:06 ` adiabat
2017-01-03 23:46 ` Aaron Voisine
2017-01-04 16:13 ` Leo Wandersleb
2017-01-04 7:47 ` Jonas Schnelli
2 siblings, 2 replies; 34+ messages in thread
From: adiabat @ 2017-01-03 23:06 UTC (permalink / raw)
To: Aaron Voisine, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 4518 bytes --]
Mempool transactions have their place, but "unconfirmed" and "SPV" don't
belong together. Only a full node can tell if a transaction may get
confirmed, or is nonsense. Unfortunately all the light / SPV wallets I
know of show mempool transactions, which makes it hard to go back... (e.g.
"why doesn't your software show 0-conf! your wallet is broken!", somewhat
akin to people complaining about RBF)
So, this is easy, just don't worry about mempool filtering. Why are light
clients looking at the mempool anyway? Maybe if there were some way to
provide SPV proofs of all inputs, but that's a bit of a mess for full nodes
to do.
Without mempool filtering, I think the committed bloom filters would be a
great improvement over the current bloom filter setup, especially for
lightning network use cases (with lightning, not finding out about a
transaction can make you lose money). I want to work on it and may be able
to at some point as it's somewhat related to lightning.
Also, if you're running a light client, and storing the filters the way you
store block headers, there's really no reason to go all the way back to
height 0. You can start grabbing headers at some point a while ago, before
your set of keys was generated. I think it'd be very worth it even with
GB-scale disk usage.
-Tadge
On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:
> Unconfirmed transactions are incredibly important for real world use.
> Merchants for instance are willing to accept credit card payments of
> thousands of dollars and ship the goods despite the fact that the
> transaction can be reversed up to 60 days later. There is a very large cost
> to losing the ability to have instant transactions in many or even most
> situations. This cost is typically well above the fraud risk.
>
> It's important to recognize that bitcoin serves a wide variety of use
> cases with different profiles for time sensitivity and fraud risk.
>
> Aaron
>
> On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev <bitcoin-dev@lists.
> linuxfoundation.org> wrote:
>
>> The concept combined with the weak blocks system where miners commit
>>
>> to potential transaction inclusion with fractional difficulty blocks
>>
>> is possible. I'm not personally convinced that unconfirmed transaction
>>
>> display in a wallet is worth the privacy trade-off. The user has very
>>
>> little to gain from this knowledge until the txn is in a block.
>>
>>
>>
>>
>>
>> On 2017-01-01 13:01, Jonas Schnelli via bitcoin-dev wrote:
>>
>> > Hi
>>
>> >> We introduce several concepts that rework the lightweight Bitcoin
>>
>> >> client model in a manner which is secure, efficient and privacy
>>
>> >> compatible.
>>
>> >>
>>
>> >> The BFD can be used verbatim in replacement of BIP37, where the filter
>>
>> >> can be cached between clients without needing to be recomputed. It can
>>
>> >> also be used by normal pruned nodes to do re-scans locally of their
>>
>> >> wallet without needing to have the block data available to scan, or
>>
>> >> without reading the entire block chain from disk.
>>
>> > I started exploring the potential of BFD after this specification.
>>
>> >
>>
>> > What would be the preferred/recommended way to handle 0-conf/mempool
>>
>> > filtering – if & once BDF would have been deployed (any type,
>>
>> > semi-trusted oracles or protocol-level/softfork)?
>>
>> >
>>
>> > From the user-experience perspective, this is probably pretty important
>>
>> > (otherwise the experience will be that incoming funds can take serval
>>
>> > minutes to hours until they appear).
>>
>> > Using BIP37 bloom filters just for mempool filtering would obviously
>>
>> > result in the same unwanted privacy-setup.
>>
>> >
>>
>> > </jonas>
>>
>> >
>>
>> >
>>
>> > _______________________________________________
>>
>> > bitcoin-dev mailing list
>>
>> > bitcoin-dev@lists•linuxfoundation.org
>>
>> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> _______________________________________________
>>
>> bitcoin-dev mailing list
>>
>> bitcoin-dev@lists•linuxfoundation.org
>>
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
[-- Attachment #2: Type: text/html, Size: 8048 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-03 23:06 ` adiabat
@ 2017-01-03 23:46 ` Aaron Voisine
2017-01-04 0:10 ` bfd
2017-01-04 16:13 ` Leo Wandersleb
1 sibling, 1 reply; 34+ messages in thread
From: Aaron Voisine @ 2017-01-03 23:46 UTC (permalink / raw)
To: adiabat; +Cc: Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 5074 bytes --]
If the sender doesn't control the receiver's network connection, then the
information the receiver gains by watching the mempool is if the
transaction has propagated across the bitcoin network. This is useful to
know in all kinds of situations.
Aaron Voisine
co-founder and CEO
breadwallet <http://breadwallet.com>
On Tue, Jan 3, 2017 at 3:06 PM, adiabat <rx@awsomnet•org> wrote:
> Mempool transactions have their place, but "unconfirmed" and "SPV" don't
> belong together. Only a full node can tell if a transaction may get
> confirmed, or is nonsense. Unfortunately all the light / SPV wallets I
> know of show mempool transactions, which makes it hard to go back... (e.g.
> "why doesn't your software show 0-conf! your wallet is broken!", somewhat
> akin to people complaining about RBF)
>
> So, this is easy, just don't worry about mempool filtering. Why are light
> clients looking at the mempool anyway? Maybe if there were some way to
> provide SPV proofs of all inputs, but that's a bit of a mess for full nodes
> to do.
>
> Without mempool filtering, I think the committed bloom filters would be a
> great improvement over the current bloom filter setup, especially for
> lightning network use cases (with lightning, not finding out about a
> transaction can make you lose money). I want to work on it and may be able
> to at some point as it's somewhat related to lightning.
>
> Also, if you're running a light client, and storing the filters the way
> you store block headers, there's really no reason to go all the way back to
> height 0. You can start grabbing headers at some point a while ago, before
> your set of keys was generated. I think it'd be very worth it even with
> GB-scale disk usage.
>
> -Tadge
>
>
> On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation.org> wrote:
>
>> Unconfirmed transactions are incredibly important for real world use.
>> Merchants for instance are willing to accept credit card payments of
>> thousands of dollars and ship the goods despite the fact that the
>> transaction can be reversed up to 60 days later. There is a very large cost
>> to losing the ability to have instant transactions in many or even most
>> situations. This cost is typically well above the fraud risk.
>>
>> It's important to recognize that bitcoin serves a wide variety of use
>> cases with different profiles for time sensitivity and fraud risk.
>>
>> Aaron
>>
>> On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev <
>> bitcoin-dev@lists•linuxfoundation.org> wrote:
>>
>>> The concept combined with the weak blocks system where miners commit
>>>
>>> to potential transaction inclusion with fractional difficulty blocks
>>>
>>> is possible. I'm not personally convinced that unconfirmed transaction
>>>
>>> display in a wallet is worth the privacy trade-off. The user has very
>>>
>>> little to gain from this knowledge until the txn is in a block.
>>>
>>>
>>>
>>>
>>>
>>> On 2017-01-01 13:01, Jonas Schnelli via bitcoin-dev wrote:
>>>
>>> > Hi
>>>
>>> >> We introduce several concepts that rework the lightweight Bitcoin
>>>
>>> >> client model in a manner which is secure, efficient and privacy
>>>
>>> >> compatible.
>>>
>>> >>
>>>
>>> >> The BFD can be used verbatim in replacement of BIP37, where the filter
>>>
>>> >> can be cached between clients without needing to be recomputed. It can
>>>
>>> >> also be used by normal pruned nodes to do re-scans locally of their
>>>
>>> >> wallet without needing to have the block data available to scan, or
>>>
>>> >> without reading the entire block chain from disk.
>>>
>>> > I started exploring the potential of BFD after this specification.
>>>
>>> >
>>>
>>> > What would be the preferred/recommended way to handle 0-conf/mempool
>>>
>>> > filtering – if & once BDF would have been deployed (any type,
>>>
>>> > semi-trusted oracles or protocol-level/softfork)?
>>>
>>> >
>>>
>>> > From the user-experience perspective, this is probably pretty important
>>>
>>> > (otherwise the experience will be that incoming funds can take serval
>>>
>>> > minutes to hours until they appear).
>>>
>>> > Using BIP37 bloom filters just for mempool filtering would obviously
>>>
>>> > result in the same unwanted privacy-setup.
>>>
>>> >
>>>
>>> > </jonas>
>>>
>>> >
>>>
>>> >
>>>
>>> > _______________________________________________
>>>
>>> > bitcoin-dev mailing list
>>>
>>> > bitcoin-dev@lists•linuxfoundation.org
>>>
>>> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>> _______________________________________________
>>>
>>> bitcoin-dev mailing list
>>>
>>> bitcoin-dev@lists•linuxfoundation.org
>>>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>
[-- Attachment #2: Type: text/html, Size: 9974 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-03 23:46 ` Aaron Voisine
@ 2017-01-04 0:10 ` bfd
2017-01-04 0:36 ` Aaron Voisine
0 siblings, 1 reply; 34+ messages in thread
From: bfd @ 2017-01-04 0:10 UTC (permalink / raw)
To: Aaron Voisine; +Cc: Bitcoin Protocol Discussion
Unfortunately a non validating SPV wallet has absolutely no idea if
the information about an unconfirmed transaction they are seeing is
anything but properly formatted. They are connecting to an easily
manipulated, sybil attacked, and untrusted network and then asking
them for financial information. Seeing an unconfirmed transaction in a
wallet that's not also fully validating is at best meaningless.
On 2017-01-03 15:46, Aaron Voisine wrote:
> If the sender doesn't control the receiver's network connection, then
> the information the receiver gains by watching the mempool is if the
> transaction has propagated across the bitcoin network. This is useful
> to know in all kinds of situations.
>
> Aaron Voisine
> co-founder and CEO
> breadwallet [2]
> On Tue, Jan 3, 2017 at 3:06 PM, adiabat <rx@awsomnet•org> wrote:
>
>> Mempool transactions have their place, but "unconfirmed" and "SPV"
>> don't belong together. Only a full node can tell if a transaction
>> may get confirmed, or is nonsense. Unfortunately all the light /
>> SPV wallets I know of show mempool transactions, which makes it hard
>> to go back... (e.g. "why doesn't your software show 0-conf! your
>> wallet is broken!", somewhat akin to people complaining about RBF)
>>
>> So, this is easy, just don't worry about mempool filtering. Why are
>> light clients looking at the mempool anyway? Maybe if there were
>> some way to provide SPV proofs of all inputs, but that's a bit of a
>> mess for full nodes to do.
>>
>> Without mempool filtering, I think the committed bloom filters would
>> be a great improvement over the current bloom filter setup,
>> especially for lightning network use cases (with lightning, not
>> finding out about a transaction can make you lose money). I want to
>> work on it and may be able to at some point as it's somewhat related
>> to lightning.
>>
>> Also, if you're running a light client, and storing the filters the
>> way you store block headers, there's really no reason to go all the
>> way back to height 0. You can start grabbing headers at some point
>> a while ago, before your set of keys was generated. I think it'd be
>> very worth it even with GB-scale disk usage.
>>
>> -Tadge
>>
>> On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev
>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>
>> Unconfirmed transactions are incredibly important for real world
>> use. Merchants for instance are willing to accept credit card
>> payments of thousands of dollars and ship the goods despite the fact
>> that the transaction can be reversed up to 60 days later. There is a
>> very large cost to losing the ability to have instant transactions
>> in many or even most situations. This cost is typically well above
>> the fraud risk.
>>
>> It's important to recognize that bitcoin serves a wide variety of
>> use cases with different profiles for time sensitivity and fraud
>> risk.
>>
>> Aaron
>>
>> On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev
>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>> The concept combined with the weak blocks system where miners commit
>>
>> to potential transaction inclusion with fractional difficulty blocks
>>
>> is possible. I'm not personally convinced that unconfirmed
>> transaction
>>
>> display in a wallet is worth the privacy trade-off. The user has
>> very
>>
>> little to gain from this knowledge until the txn is in a block.
>>
>> On 2017-01-01 13:01, Jonas Schnelli via bitcoin-dev wrote:
>>
>>> Hi
>>
>>>> We introduce several concepts that rework the lightweight Bitcoin
>>
>>>> client model in a manner which is secure, efficient and privacy
>>
>>>> compatible.
>>
>>>>
>>
>>>> The BFD can be used verbatim in replacement of BIP37, where the
>> filter
>>
>>>> can be cached between clients without needing to be recomputed.
>> It can
>>
>>>> also be used by normal pruned nodes to do re-scans locally of
>> their
>>
>>>> wallet without needing to have the block data available to scan,
>> or
>>
>>>> without reading the entire block chain from disk.
>>
>>> I started exploring the potential of BFD after this specification.
>>
>>>
>>
>>> What would be the preferred/recommended way to handle
>> 0-conf/mempool
>>
>>> filtering – if & once BDF would have been deployed (any type,
>>
>>> semi-trusted oracles or protocol-level/softfork)?
>>
>>>
>>
>>> From the user-experience perspective, this is probably pretty
>> important
>>
>>> (otherwise the experience will be that incoming funds can take
>> serval
>>
>>> minutes to hours until they appear).
>>
>>> Using BIP37 bloom filters just for mempool filtering would
>> obviously
>>
>>> result in the same unwanted privacy-setup.
>>
>>>
>>
>>> </jonas>
>>
>>>
>>
>>>
>>
>>> _______________________________________________
>>
>>> bitcoin-dev mailing list
>>
>>> bitcoin-dev@lists•linuxfoundation.org
>>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>>
>> _______________________________________________
>>
>> bitcoin-dev mailing list
>>
>> bitcoin-dev@lists•linuxfoundation.org
>>
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>
>
>
> Links:
> ------
> [1] https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> [2] http://breadwallet.com
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-04 0:10 ` bfd
@ 2017-01-04 0:36 ` Aaron Voisine
2017-01-04 6:06 ` Eric Voskuil
0 siblings, 1 reply; 34+ messages in thread
From: Aaron Voisine @ 2017-01-04 0:36 UTC (permalink / raw)
To: bfd; +Cc: Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 6542 bytes --]
Knowing that a transaction is property formatted and that it has been
broadcast to the gossip network is useful in many situations. You're only
thinking about whether you can know a transaction is valid and/or settled.
This is not the only possible useful information in actual real world use.
Any situation where credit card transactions are accepted today for
instance, it is useful to know that a transaction has been initiated, even
though it can be reversed at any time up to 60 days later.
Aaron Voisine
co-founder and CEO
breadwallet <http://breadwallet.com>
On Tue, Jan 3, 2017 at 4:10 PM, <bfd@cock•lu> wrote:
> Unfortunately a non validating SPV wallet has absolutely no idea if
> the information about an unconfirmed transaction they are seeing is
> anything but properly formatted. They are connecting to an easily
> manipulated, sybil attacked, and untrusted network and then asking
> them for financial information. Seeing an unconfirmed transaction in a
> wallet that's not also fully validating is at best meaningless.
>
>
> On 2017-01-03 15:46, Aaron Voisine wrote:
>
>> If the sender doesn't control the receiver's network connection, then
>> the information the receiver gains by watching the mempool is if the
>> transaction has propagated across the bitcoin network. This is useful
>> to know in all kinds of situations.
>>
>> Aaron Voisine
>> co-founder and CEO
>> breadwallet [2]
>>
>> On Tue, Jan 3, 2017 at 3:06 PM, adiabat <rx@awsomnet•org> wrote:
>>
>> Mempool transactions have their place, but "unconfirmed" and "SPV"
>>> don't belong together. Only a full node can tell if a transaction
>>> may get confirmed, or is nonsense. Unfortunately all the light /
>>> SPV wallets I know of show mempool transactions, which makes it hard
>>> to go back... (e.g. "why doesn't your software show 0-conf! your
>>> wallet is broken!", somewhat akin to people complaining about RBF)
>>>
>>> So, this is easy, just don't worry about mempool filtering. Why are
>>> light clients looking at the mempool anyway? Maybe if there were
>>> some way to provide SPV proofs of all inputs, but that's a bit of a
>>> mess for full nodes to do.
>>>
>>> Without mempool filtering, I think the committed bloom filters would
>>> be a great improvement over the current bloom filter setup,
>>> especially for lightning network use cases (with lightning, not
>>> finding out about a transaction can make you lose money). I want to
>>> work on it and may be able to at some point as it's somewhat related
>>> to lightning.
>>>
>>> Also, if you're running a light client, and storing the filters the
>>> way you store block headers, there's really no reason to go all the
>>> way back to height 0. You can start grabbing headers at some point
>>> a while ago, before your set of keys was generated. I think it'd be
>>> very worth it even with GB-scale disk usage.
>>>
>>> -Tadge
>>>
>>> On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev
>>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>>
>>> Unconfirmed transactions are incredibly important for real world
>>> use. Merchants for instance are willing to accept credit card
>>> payments of thousands of dollars and ship the goods despite the fact
>>> that the transaction can be reversed up to 60 days later. There is a
>>> very large cost to losing the ability to have instant transactions
>>> in many or even most situations. This cost is typically well above
>>> the fraud risk.
>>>
>>> It's important to recognize that bitcoin serves a wide variety of
>>> use cases with different profiles for time sensitivity and fraud
>>> risk.
>>>
>>> Aaron
>>>
>>> On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev
>>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>> The concept combined with the weak blocks system where miners commit
>>>
>>> to potential transaction inclusion with fractional difficulty blocks
>>>
>>> is possible. I'm not personally convinced that unconfirmed
>>> transaction
>>>
>>> display in a wallet is worth the privacy trade-off. The user has
>>> very
>>>
>>> little to gain from this knowledge until the txn is in a block.
>>>
>>> On 2017-01-01 13:01, Jonas Schnelli via bitcoin-dev wrote:
>>>
>>> Hi
>>>>
>>>
>>> We introduce several concepts that rework the lightweight Bitcoin
>>>>>
>>>>
>>> client model in a manner which is secure, efficient and privacy
>>>>>
>>>>
>>> compatible.
>>>>>
>>>>
>>>
>>>>>
>>> The BFD can be used verbatim in replacement of BIP37, where the
>>>>>
>>>> filter
>>>
>>> can be cached between clients without needing to be recomputed.
>>>>>
>>>> It can
>>>
>>> also be used by normal pruned nodes to do re-scans locally of
>>>>>
>>>> their
>>>
>>> wallet without needing to have the block data available to scan,
>>>>>
>>>> or
>>>
>>> without reading the entire block chain from disk.
>>>>>
>>>>
>>> I started exploring the potential of BFD after this specification.
>>>>
>>>
>>>
>>>>
>>> What would be the preferred/recommended way to handle
>>>>
>>> 0-conf/mempool
>>>
>>> filtering – if & once BDF would have been deployed (any type,
>>>>
>>>
>>> semi-trusted oracles or protocol-level/softfork)?
>>>>
>>>
>>>
>>>>
>>> From the user-experience perspective, this is probably pretty
>>>>
>>> important
>>>
>>> (otherwise the experience will be that incoming funds can take
>>>>
>>> serval
>>>
>>> minutes to hours until they appear).
>>>>
>>>
>>> Using BIP37 bloom filters just for mempool filtering would
>>>>
>>> obviously
>>>
>>> result in the same unwanted privacy-setup.
>>>>
>>>
>>>
>>>>
>>> </jonas>
>>>>
>>>
>>>
>>>>
>>>
>>>>
>>> _______________________________________________
>>>>
>>>
>>> bitcoin-dev mailing list
>>>>
>>>
>>> bitcoin-dev@lists•linuxfoundation.org
>>>>
>>>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>>>>
>>>
>>> _______________________________________________
>>>
>>> bitcoin-dev mailing list
>>>
>>> bitcoin-dev@lists•linuxfoundation.org
>>>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists•linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>>>
>>
>>
>>
>> Links:
>> ------
>> [1] https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> [2] http://breadwallet.com
>>
>
[-- Attachment #2: Type: text/html, Size: 12973 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-04 0:36 ` Aaron Voisine
@ 2017-01-04 6:06 ` Eric Voskuil
0 siblings, 0 replies; 34+ messages in thread
From: Eric Voskuil @ 2017-01-04 6:06 UTC (permalink / raw)
To: Aaron Voisine, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 7244 bytes --]
Credit card reversals involve an escrow agent with control over the entire network and with a strong interest in preserving the network. A better analogy would be blind acceptance of any slip of paper under the assumption that it is sufficient currency. It may or may not be so, but you are on your own in either case.
e
> On Jan 3, 2017, at 4:36 PM, Aaron Voisine via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
>
> Knowing that a transaction is property formatted and that it has been broadcast to the gossip network is useful in many situations. You're only thinking about whether you can know a transaction is valid and/or settled. This is not the only possible useful information in actual real world use. Any situation where credit card transactions are accepted today for instance, it is useful to know that a transaction has been initiated, even though it can be reversed at any time up to 60 days later.
>
> Aaron Voisine
> co-founder and CEO
> breadwallet
>
>> On Tue, Jan 3, 2017 at 4:10 PM, <bfd@cock•lu> wrote:
>> Unfortunately a non validating SPV wallet has absolutely no idea if
>> the information about an unconfirmed transaction they are seeing is
>> anything but properly formatted. They are connecting to an easily
>> manipulated, sybil attacked, and untrusted network and then asking
>> them for financial information. Seeing an unconfirmed transaction in a
>> wallet that's not also fully validating is at best meaningless.
>>
>>
>>> On 2017-01-03 15:46, Aaron Voisine wrote:
>>> If the sender doesn't control the receiver's network connection, then
>>> the information the receiver gains by watching the mempool is if the
>>> transaction has propagated across the bitcoin network. This is useful
>>> to know in all kinds of situations.
>>>
>>> Aaron Voisine
>>> co-founder and CEO
>>> breadwallet [2]
>>>
>>> On Tue, Jan 3, 2017 at 3:06 PM, adiabat <rx@awsomnet•org> wrote:
>>>
>>>> Mempool transactions have their place, but "unconfirmed" and "SPV"
>>>> don't belong together. Only a full node can tell if a transaction
>>>> may get confirmed, or is nonsense. Unfortunately all the light /
>>>> SPV wallets I know of show mempool transactions, which makes it hard
>>>> to go back... (e.g. "why doesn't your software show 0-conf! your
>>>> wallet is broken!", somewhat akin to people complaining about RBF)
>>>>
>>>> So, this is easy, just don't worry about mempool filtering. Why are
>>>> light clients looking at the mempool anyway? Maybe if there were
>>>> some way to provide SPV proofs of all inputs, but that's a bit of a
>>>> mess for full nodes to do.
>>>>
>>>> Without mempool filtering, I think the committed bloom filters would
>>>> be a great improvement over the current bloom filter setup,
>>>> especially for lightning network use cases (with lightning, not
>>>> finding out about a transaction can make you lose money). I want to
>>>> work on it and may be able to at some point as it's somewhat related
>>>> to lightning.
>>>>
>>>> Also, if you're running a light client, and storing the filters the
>>>> way you store block headers, there's really no reason to go all the
>>>> way back to height 0. You can start grabbing headers at some point
>>>> a while ago, before your set of keys was generated. I think it'd be
>>>> very worth it even with GB-scale disk usage.
>>>>
>>>> -Tadge
>>>>
>>>> On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev
>>>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>>>
>>>> Unconfirmed transactions are incredibly important for real world
>>>> use. Merchants for instance are willing to accept credit card
>>>> payments of thousands of dollars and ship the goods despite the fact
>>>> that the transaction can be reversed up to 60 days later. There is a
>>>> very large cost to losing the ability to have instant transactions
>>>> in many or even most situations. This cost is typically well above
>>>> the fraud risk.
>>>>
>>>> It's important to recognize that bitcoin serves a wide variety of
>>>> use cases with different profiles for time sensitivity and fraud
>>>> risk.
>>>>
>>>> Aaron
>>>>
>>>> On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev
>>>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>>> The concept combined with the weak blocks system where miners commit
>>>>
>>>> to potential transaction inclusion with fractional difficulty blocks
>>>>
>>>> is possible. I'm not personally convinced that unconfirmed
>>>> transaction
>>>>
>>>> display in a wallet is worth the privacy trade-off. The user has
>>>> very
>>>>
>>>> little to gain from this knowledge until the txn is in a block.
>>>>
>>>> On 2017-01-01 13:01, Jonas Schnelli via bitcoin-dev wrote:
>>>>
>>>>> Hi
>>>>
>>>>>> We introduce several concepts that rework the lightweight Bitcoin
>>>>
>>>>>> client model in a manner which is secure, efficient and privacy
>>>>
>>>>>> compatible.
>>>>
>>>>>>
>>>>
>>>>>> The BFD can be used verbatim in replacement of BIP37, where the
>>>> filter
>>>>
>>>>>> can be cached between clients without needing to be recomputed.
>>>> It can
>>>>
>>>>>> also be used by normal pruned nodes to do re-scans locally of
>>>> their
>>>>
>>>>>> wallet without needing to have the block data available to scan,
>>>> or
>>>>
>>>>>> without reading the entire block chain from disk.
>>>>
>>>>> I started exploring the potential of BFD after this specification.
>>>>
>>>>>
>>>>
>>>>> What would be the preferred/recommended way to handle
>>>> 0-conf/mempool
>>>>
>>>>> filtering – if & once BDF would have been deployed (any type,
>>>>
>>>>> semi-trusted oracles or protocol-level/softfork)?
>>>>
>>>>>
>>>>
>>>>> From the user-experience perspective, this is probably pretty
>>>> important
>>>>
>>>>> (otherwise the experience will be that incoming funds can take
>>>> serval
>>>>
>>>>> minutes to hours until they appear).
>>>>
>>>>> Using BIP37 bloom filters just for mempool filtering would
>>>> obviously
>>>>
>>>>> result in the same unwanted privacy-setup.
>>>>
>>>>>
>>>>
>>>>> </jonas>
>>>>
>>>>>
>>>>
>>>>>
>>>>
>>>>> _______________________________________________
>>>>
>>>>> bitcoin-dev mailing list
>>>>
>>>>> bitcoin-dev@lists•linuxfoundation.org
>>>>
>>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>>>>
>>>> _______________________________________________
>>>>
>>>> bitcoin-dev mailing list
>>>>
>>>> bitcoin-dev@lists•linuxfoundation.org
>>>>
>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>>>>
>>>> _______________________________________________
>>>> bitcoin-dev mailing list
>>>> bitcoin-dev@lists•linuxfoundation.org
>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
>>>
>>>
>>>
>>> Links:
>>> ------
>>> [1] https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>> [2] http://breadwallet.com
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[-- Attachment #2: Type: text/html, Size: 14061 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-03 22:18 ` Aaron Voisine
2017-01-03 22:28 ` bfd
2017-01-03 23:06 ` adiabat
@ 2017-01-04 7:47 ` Jonas Schnelli
2017-01-04 8:56 ` Aaron Voisine
2017-01-05 7:06 ` Chris Priest
2 siblings, 2 replies; 34+ messages in thread
From: Jonas Schnelli @ 2017-01-04 7:47 UTC (permalink / raw)
To: Aaron Voisine, Bitcoin Protocol Discussion, bfd
[-- Attachment #1.1: Type: text/plain, Size: 2215 bytes --]
Hi
> Unconfirmed transactions are incredibly important for real world use.
> Merchants for instance are willing to accept credit card payments of
> thousands of dollars and ship the goods despite the fact that the
> transaction can be reversed up to 60 days later. There is a very large
> cost to losing the ability to have instant transactions in many or
> even most situations. This cost is typically well above the fraud risk.
>
> It's important to recognize that bitcoin serves a wide variety of use
> cases with different profiles for time sensitivity and fraud risk.
>
I agree that unconfirmed transactions are incredibly important, but not
over SPV against random peers.
If you offer users/merchants a feature (SPV 0-conf against random
peers), that is fundamentally insecure, it will – sooner or later – lead
to some large scale fiasco, hurting Bitcoins reputation and trust from
merchants.
Merchants using and trusting 0-conf SPV transactions (retrieved from
random peers) is something we should **really eliminate** through
education and by offering different solution.
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
For end-users SPV software, I think it would be recommended to...
... disable unconfirmed transactions during SPV against random peers
... enable unconfirmed transactions when using SPV against a trusted
peer with preshared keys after BIP150
... if unconfirmed transactions are disabled, show how it can be enabled
(how to run a full-node [in a box, etc.])
... educate, inform users that a transaction with no confirmation can be
"stopped" or "redirected" any time, also inform about the risks during
low-conf phase (1-5).
I though see the point that it's nice to make use of the "incoming
funds..." feature in SPV wallets. But – for the sake of stability and
(risk-)scaling – we may want to recommend to scarify this feature and –
in the same turn – to use privacy-preserving BFD's.
</jonas>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-04 7:47 ` Jonas Schnelli
@ 2017-01-04 8:56 ` Aaron Voisine
2017-01-04 10:13 ` Jorge Timón
2017-01-06 2:15 ` bfd
2017-01-05 7:06 ` Chris Priest
1 sibling, 2 replies; 34+ messages in thread
From: Aaron Voisine @ 2017-01-04 8:56 UTC (permalink / raw)
To: Bitcoin Protocol Discussion, Jonas Schnelli, bfd
[-- Attachment #1: Type: text/plain, Size: 3274 bytes --]
It's easy enough to mark a transaction as "pending". People with bank
accounts are familiar with the concept.
Although the risk of accepting gossip information from multiple random
peers, in the case where the sender does not control the receivers network
is still minimal. Random node operators have no incentive to send fake
transactions, and would need to control all the nodes a client connects to,
and find a non-false-positive address belonging to the victims wallet.
It's not impossible, but it's non trivial, would only temporarily show a
pending transaction, and provide no benefit to the node operator. There are
much juicier targets for an attacker with the ability to sybil attack the
entire bitcoin p2p network.
Aaron
On Tue, Jan 3, 2017 at 11:47 PM Jonas Schnelli <dev@jonasschnelli•ch> wrote:
> Hi
>
>
>
> > Unconfirmed transactions are incredibly important for real world use.
>
> > Merchants for instance are willing to accept credit card payments of
>
> > thousands of dollars and ship the goods despite the fact that the
>
> > transaction can be reversed up to 60 days later. There is a very large
>
> > cost to losing the ability to have instant transactions in many or
>
> > even most situations. This cost is typically well above the fraud risk.
>
> >
>
> > It's important to recognize that bitcoin serves a wide variety of use
>
> > cases with different profiles for time sensitivity and fraud risk.
>
> >
>
> I agree that unconfirmed transactions are incredibly important, but not
>
> over SPV against random peers.
>
>
>
> If you offer users/merchants a feature (SPV 0-conf against random
>
> peers), that is fundamentally insecure, it will – sooner or later – lead
>
> to some large scale fiasco, hurting Bitcoins reputation and trust from
>
> merchants.
>
>
>
> Merchants using and trusting 0-conf SPV transactions (retrieved from
>
> random peers) is something we should **really eliminate** through
>
> education and by offering different solution.
>
>
>
> There are plenty, more sane options. If you can't run your own full-node
>
> as a merchant (trivial), maybe co-use a wallet-service with centralized
>
> verification (maybe use two of them), I guess Copay would be one of
>
> those wallets (as an example). Use them in watch-only mode.
>
>
>
> For end-users SPV software, I think it would be recommended to...
>
> ... disable unconfirmed transactions during SPV against random peers
>
> ... enable unconfirmed transactions when using SPV against a trusted
>
> peer with preshared keys after BIP150
>
> ... if unconfirmed transactions are disabled, show how it can be enabled
>
> (how to run a full-node [in a box, etc.])
>
> ... educate, inform users that a transaction with no confirmation can be
>
> "stopped" or "redirected" any time, also inform about the risks during
>
> low-conf phase (1-5).
>
>
>
> I though see the point that it's nice to make use of the "incoming
>
> funds..." feature in SPV wallets. But – for the sake of stability and
>
> (risk-)scaling – we may want to recommend to scarify this feature and –
>
> in the same turn – to use privacy-preserving BFD's.
>
>
>
> </jonas>
>
>
>
>
>
>
[-- Attachment #2: Type: text/html, Size: 4566 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-04 8:56 ` Aaron Voisine
@ 2017-01-04 10:13 ` Jorge Timón
2017-01-04 11:00 ` Adam Back
2017-01-06 2:15 ` bfd
1 sibling, 1 reply; 34+ messages in thread
From: Jorge Timón @ 2017-01-04 10:13 UTC (permalink / raw)
To: Aaron Voisine, Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 4206 bytes --]
There were talks about implementing spv mode for bitcoin core without using
bloom filters. Less efficient because it downloads full blocks, but better
for privacy. Perhaps other spv implementations should consider doing the
same instead of committing the filters in the block?
Now I feel I was missing something. I guess you can download the whole
block you're interested in instead of only your txs and that gives you
privacy.
But how do you get to know which blocks are you interested in?
If the questions are too basic or offtopic for the thread, I'm happy
getting answers privately (but then maybe I get them more than once).
On 4 Jan 2017 09:57, "Aaron Voisine via bitcoin-dev" <
bitcoin-dev@lists•linuxfoundation.org> wrote:
It's easy enough to mark a transaction as "pending". People with bank
accounts are familiar with the concept.
Although the risk of accepting gossip information from multiple random
peers, in the case where the sender does not control the receivers network
is still minimal. Random node operators have no incentive to send fake
transactions, and would need to control all the nodes a client connects to,
and find a non-false-positive address belonging to the victims wallet.
It's not impossible, but it's non trivial, would only temporarily show a
pending transaction, and provide no benefit to the node operator. There are
much juicier targets for an attacker with the ability to sybil attack the
entire bitcoin p2p network.
Aaron
On Tue, Jan 3, 2017 at 11:47 PM Jonas Schnelli <dev@jonasschnelli•ch> wrote:
> Hi
>
>
>
> > Unconfirmed transactions are incredibly important for real world use.
>
> > Merchants for instance are willing to accept credit card payments of
>
> > thousands of dollars and ship the goods despite the fact that the
>
> > transaction can be reversed up to 60 days later. There is a very large
>
> > cost to losing the ability to have instant transactions in many or
>
> > even most situations. This cost is typically well above the fraud risk.
>
> >
>
> > It's important to recognize that bitcoin serves a wide variety of use
>
> > cases with different profiles for time sensitivity and fraud risk.
>
> >
>
> I agree that unconfirmed transactions are incredibly important, but not
>
> over SPV against random peers.
>
>
>
> If you offer users/merchants a feature (SPV 0-conf against random
>
> peers), that is fundamentally insecure, it will – sooner or later – lead
>
> to some large scale fiasco, hurting Bitcoins reputation and trust from
>
> merchants.
>
>
>
> Merchants using and trusting 0-conf SPV transactions (retrieved from
>
> random peers) is something we should **really eliminate** through
>
> education and by offering different solution.
>
>
>
> There are plenty, more sane options. If you can't run your own full-node
>
> as a merchant (trivial), maybe co-use a wallet-service with centralized
>
> verification (maybe use two of them), I guess Copay would be one of
>
> those wallets (as an example). Use them in watch-only mode.
>
>
>
> For end-users SPV software, I think it would be recommended to...
>
> ... disable unconfirmed transactions during SPV against random peers
>
> ... enable unconfirmed transactions when using SPV against a trusted
>
> peer with preshared keys after BIP150
>
> ... if unconfirmed transactions are disabled, show how it can be enabled
>
> (how to run a full-node [in a box, etc.])
>
> ... educate, inform users that a transaction with no confirmation can be
>
> "stopped" or "redirected" any time, also inform about the risks during
>
> low-conf phase (1-5).
>
>
>
> I though see the point that it's nice to make use of the "incoming
>
> funds..." feature in SPV wallets. But – for the sake of stability and
>
> (risk-)scaling – we may want to recommend to scarify this feature and –
>
> in the same turn – to use privacy-preserving BFD's.
>
>
>
> </jonas>
>
>
>
>
>
>
_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists•linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[-- Attachment #2: Type: text/html, Size: 7207 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-04 10:13 ` Jorge Timón
@ 2017-01-04 11:00 ` Adam Back
0 siblings, 0 replies; 34+ messages in thread
From: Adam Back @ 2017-01-04 11:00 UTC (permalink / raw)
To: Jorge Timón, Bitcoin Protocol Discussion
I think this discussion started from the block bloom filter where
there is a bloom filter commitment in the block which can be
downloaded and is much smaller than the block. An SPV node based on
that model would download headers and bloom filters, verify the bloom
filter is committed to, and test locally if any addresses managed by
the wallet are in the filter (or false positives for being in it), and
then download blocks with hits. Apparently there are maybe 50% more
compact alternatives to bloom filters but people have been using bloom
filter as a short-hand for that. The block bloom filter does seem to
have higher overhead than the query model, but it offers much better
privacy. I think there was previous discussion about maybe doing
something with portions of blocks so you can know which half or
quarter of the block etc.
Adam
On 4 January 2017 at 10:13, Jorge Timón via bitcoin-dev
<bitcoin-dev@lists•linuxfoundation.org> wrote:
> There were talks about implementing spv mode for bitcoin core without using
> bloom filters. Less efficient because it downloads full blocks, but better
> for privacy. Perhaps other spv implementations should consider doing the
> same instead of committing the filters in the block?
>
> Now I feel I was missing something. I guess you can download the whole block
> you're interested in instead of only your txs and that gives you privacy.
> But how do you get to know which blocks are you interested in?
>
> If the questions are too basic or offtopic for the thread, I'm happy getting
> answers privately (but then maybe I get them more than once).
>
>
> On 4 Jan 2017 09:57, "Aaron Voisine via bitcoin-dev"
> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>
> It's easy enough to mark a transaction as "pending". People with bank
> accounts are familiar with the concept.
>
> Although the risk of accepting gossip information from multiple random
> peers, in the case where the sender does not control the receivers network
> is still minimal. Random node operators have no incentive to send fake
> transactions, and would need to control all the nodes a client connects to,
> and find a non-false-positive address belonging to the victims wallet.
>
> It's not impossible, but it's non trivial, would only temporarily show a
> pending transaction, and provide no benefit to the node operator. There are
> much juicier targets for an attacker with the ability to sybil attack the
> entire bitcoin p2p network.
>
> Aaron
>
> On Tue, Jan 3, 2017 at 11:47 PM Jonas Schnelli <dev@jonasschnelli•ch> wrote:
>>
>> Hi
>>
>>
>>
>> > Unconfirmed transactions are incredibly important for real world use.
>>
>> > Merchants for instance are willing to accept credit card payments of
>>
>> > thousands of dollars and ship the goods despite the fact that the
>>
>> > transaction can be reversed up to 60 days later. There is a very large
>>
>> > cost to losing the ability to have instant transactions in many or
>>
>> > even most situations. This cost is typically well above the fraud risk.
>>
>> >
>>
>> > It's important to recognize that bitcoin serves a wide variety of use
>>
>> > cases with different profiles for time sensitivity and fraud risk.
>>
>> >
>>
>> I agree that unconfirmed transactions are incredibly important, but not
>>
>> over SPV against random peers.
>>
>>
>>
>> If you offer users/merchants a feature (SPV 0-conf against random
>>
>> peers), that is fundamentally insecure, it will – sooner or later – lead
>>
>> to some large scale fiasco, hurting Bitcoins reputation and trust from
>>
>> merchants.
>>
>>
>>
>> Merchants using and trusting 0-conf SPV transactions (retrieved from
>>
>> random peers) is something we should **really eliminate** through
>>
>> education and by offering different solution.
>>
>>
>>
>> There are plenty, more sane options. If you can't run your own full-node
>>
>> as a merchant (trivial), maybe co-use a wallet-service with centralized
>>
>> verification (maybe use two of them), I guess Copay would be one of
>>
>> those wallets (as an example). Use them in watch-only mode.
>>
>>
>>
>> For end-users SPV software, I think it would be recommended to...
>>
>> ... disable unconfirmed transactions during SPV against random peers
>>
>> ... enable unconfirmed transactions when using SPV against a trusted
>>
>> peer with preshared keys after BIP150
>>
>> ... if unconfirmed transactions are disabled, show how it can be enabled
>>
>> (how to run a full-node [in a box, etc.])
>>
>> ... educate, inform users that a transaction with no confirmation can be
>>
>> "stopped" or "redirected" any time, also inform about the risks during
>>
>> low-conf phase (1-5).
>>
>>
>>
>> I though see the point that it's nice to make use of the "incoming
>>
>> funds..." feature in SPV wallets. But – for the sake of stability and
>>
>> (risk-)scaling – we may want to recommend to scarify this feature and –
>>
>> in the same turn – to use privacy-preserving BFD's.
>>
>>
>>
>> </jonas>
>>
>>
>>
>>
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-03 23:06 ` adiabat
2017-01-03 23:46 ` Aaron Voisine
@ 2017-01-04 16:13 ` Leo Wandersleb
1 sibling, 0 replies; 34+ messages in thread
From: Leo Wandersleb @ 2017-01-04 16:13 UTC (permalink / raw)
To: bitcoin-dev
[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]
On 01/04/2017 12:06 AM, adiabat via bitcoin-dev wrote:
> Also, if you're running a light client, and storing the filters the way you
> store block headers, there's really no reason to go all the way back to height
> 0. You can start grabbing headers at some point a while ago, before your set
> of keys was generated. I think it'd be very worth it even with GB-scale disk
> usage.
The really great benefit of having this index is that you could implement rather
efficient cold wallet spending once the wallet has the full index.
With Mycelium you can currently spend funds from a paper wallet or a BIP39
sentence but at the cost of sharing all addresses with our servers. Schildbach
would share addresses with random full nodes for hours or days to find funds of
a new private key with unknown creation date. With CBF it would still be a
matter of maybe minutes on a phone to identify relevant blocks and download
these but it would be very feasible to implement a private, cold storage
spending feature.
Also the index could be further partitioned by P2PKH, P2PK, P2SH, … This would
lead to a very minor privacy leak for a reasonable reduction in index size.
--
Leo Wandersleb
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-04 7:47 ` Jonas Schnelli
2017-01-04 8:56 ` Aaron Voisine
@ 2017-01-05 7:06 ` Chris Priest
2017-01-05 7:45 ` Eric Voskuil
2017-01-06 2:04 ` bfd
1 sibling, 2 replies; 34+ messages in thread
From: Chris Priest @ 2017-01-05 7:06 UTC (permalink / raw)
To: Jonas Schnelli, Bitcoin Protocol Discussion
On 1/3/17, Jonas Schnelli via bitcoin-dev
<bitcoin-dev@lists•linuxfoundation.org> wrote:
>
> There are plenty, more sane options. If you can't run your own full-node
> as a merchant (trivial), maybe co-use a wallet-service with centralized
> verification (maybe use two of them), I guess Copay would be one of
> those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm. If more people mined, this method will not work as well
because it would require you to call the API of hundreds of different
potential block creators.
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-05 7:06 ` Chris Priest
@ 2017-01-05 7:45 ` Eric Voskuil
2017-01-05 14:48 ` Christian Decker
2017-01-06 20:15 ` Chris Priest
2017-01-06 2:04 ` bfd
1 sibling, 2 replies; 34+ messages in thread
From: Eric Voskuil @ 2017-01-05 7:45 UTC (permalink / raw)
To: Chris Priest, Bitcoin Protocol Discussion, Jonas Schnelli
[-- Attachment #1: Type: text/plain, Size: 1617 bytes --]
On 01/04/2017 11:06 PM, Chris Priest via bitcoin-dev wrote:
> On 1/3/17, Jonas Schnelli via bitcoin-dev
> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>
>> There are plenty, more sane options. If you can't run your own full-node
>> as a merchant (trivial), maybe co-use a wallet-service with centralized
>> verification (maybe use two of them), I guess Copay would be one of
>> those wallets (as an example). Use them in watch-only mode.
>
> The best way is to connect to the mempool of each miner and check to
> see if they have your txid in their mempool.
>
> https://www.antpool.com/api/is_in_mempool?txid=334847bb...
> https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
> https://bw.com/api/is_in_mempool?txid=334847bb...
> https://bitfury.com/api/is_in_mempool?txid=334847bb...
> https://btcc.com/api/is_in_mempool?txid=334847bb...
>
> If each of these services return "True", and you know those services
> so not engage in RBF, then you can assume with great confidence that
> your transaction will be in the next block, or in a block very soon.
> If any one of those services return "False", then you must assume that
> it is possible that there is a double spend floating around, and that
> you should wait to see if that tx gets confirmed. The problem is that
> not every pool runs such a service to check the contents of their
> mempool...
>
> This is an example of mining centralization increasing the security of
> zero confirm.
A world connected up to a few web services to determine payment validity
is an example of a bitcoin security catastrophe.
e
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-05 7:45 ` Eric Voskuil
@ 2017-01-05 14:48 ` Christian Decker
2017-01-06 20:15 ` Chris Priest
1 sibling, 0 replies; 34+ messages in thread
From: Christian Decker @ 2017-01-05 14:48 UTC (permalink / raw)
To: bitcoin-dev
On Wed, Jan 04, 2017 at 11:45:18PM -0800, Eric Voskuil via bitcoin-dev wrote:
> On 01/04/2017 11:06 PM, Chris Priest via bitcoin-dev wrote:
> > The best way is to connect to the mempool of each miner and check to
> > see if they have your txid in their mempool.
> >
> > https://www.antpool.com/api/is_in_mempool?txid=334847bb...
> > https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
> > https://bw.com/api/is_in_mempool?txid=334847bb...
> > https://bitfury.com/api/is_in_mempool?txid=334847bb...
> > https://btcc.com/api/is_in_mempool?txid=334847bb...
> >
> > If each of these services return "True", and you know those services
> > so not engage in RBF, then you can assume with great confidence that
> > your transaction will be in the next block, or in a block very soon.
> > If any one of those services return "False", then you must assume that
> > it is possible that there is a double spend floating around, and that
> > you should wait to see if that tx gets confirmed. The problem is that
> > not every pool runs such a service to check the contents of their
> > mempool...
> >
> > This is an example of mining centralization increasing the security of
> > zero confirm.
>
> A world connected up to a few web services to determine payment validity
> is an example of a bitcoin security catastrophe.
>
> e
>
And it's a great way to tell every miner who you are and what
transactions you are sending/receiving. An absolute privacy
nightmare...
-- cdecker
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-05 7:06 ` Chris Priest
2017-01-05 7:45 ` Eric Voskuil
@ 2017-01-06 2:04 ` bfd
2017-03-15 22:36 ` Tom Harding
1 sibling, 1 reply; 34+ messages in thread
From: bfd @ 2017-01-06 2:04 UTC (permalink / raw)
To: Chris Priest; +Cc: Bitcoin Protocol Discussion, nbvfour
You might as well replace Bitcoin with a system where these parties
sign transactions and skip mining altogether, it would have the same
properties and be significantly more effient.
On 2017-01-04 23:06, Chris Priest wrote:
> On 1/3/17, Jonas Schnelli via bitcoin-dev
> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>
>> There are plenty, more sane options. If you can't run your own
>> full-node
>> as a merchant (trivial), maybe co-use a wallet-service with
>> centralized
>> verification (maybe use two of them), I guess Copay would be one of
>> those wallets (as an example). Use them in watch-only mode.
>
> The best way is to connect to the mempool of each miner and check to
> see if they have your txid in their mempool.
>
> https://www.antpool.com/api/is_in_mempool?txid=334847bb...
> https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
> https://bw.com/api/is_in_mempool?txid=334847bb...
> https://bitfury.com/api/is_in_mempool?txid=334847bb...
> https://btcc.com/api/is_in_mempool?txid=334847bb...
>
> If each of these services return "True", and you know those services
> so not engage in RBF, then you can assume with great confidence that
> your transaction will be in the next block, or in a block very soon.
> If any one of those services return "False", then you must assume that
> it is possible that there is a double spend floating around, and that
> you should wait to see if that tx gets confirmed. The problem is that
> not every pool runs such a service to check the contents of their
> mempool...
>
> This is an example of mining centralization increasing the security of
> zero confirm. If more people mined, this method will not work as well
> because it would require you to call the API of hundreds of different
> potential block creators.
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-04 8:56 ` Aaron Voisine
2017-01-04 10:13 ` Jorge Timón
@ 2017-01-06 2:15 ` bfd
2017-01-06 7:07 ` Aaron Voisine
1 sibling, 1 reply; 34+ messages in thread
From: bfd @ 2017-01-06 2:15 UTC (permalink / raw)
To: Aaron Voisine; +Cc: Bitcoin Protocol Discussion
With a credit card you have an institution worth billions of dollars
asserting that a payment has been made, with the option that it may be
retracted under special circumstances by the card issuer.
Unconfirmed Bitcoin transactions with a SPV client has you trusting
that the un-authenticated DNS seed lookup has not been tampered with,
the connection to the random node that you connect to has not been
tampered with, and that the seed nor the node are attempting to
manipulate you.
The two scenarios aren’t even remotely comparable.
On 2017-01-04 00:56, Aaron Voisine wrote:
> It's easy enough to mark a transaction as "pending". People with bank
> accounts are familiar with the concept.
>
> Although the risk of accepting gossip information from multiple random
> peers, in the case where the sender does not control the receivers
> network is still minimal. Random node operators have no incentive to
> send fake transactions, and would need to control all the nodes a
> client connects to, and find a non-false-positive address belonging to
> the victims wallet.
>
> It's not impossible, but it's non trivial, would only temporarily show
> a pending transaction, and provide no benefit to the node operator.
> There are much juicier targets for an attacker with the ability to
> sybil attack the entire bitcoin p2p network.
>
> Aaron
>
> On Tue, Jan 3, 2017 at 11:47 PM Jonas Schnelli <dev@jonasschnelli•ch>
> wrote:
>
>> Hi
>>
>>> Unconfirmed transactions are incredibly important for real world
>> use.
>>
>>> Merchants for instance are willing to accept credit card payments
>> of
>>
>>> thousands of dollars and ship the goods despite the fact that the
>>
>>> transaction can be reversed up to 60 days later. There is a very
>> large
>>
>>> cost to losing the ability to have instant transactions in many or
>>
>>> even most situations. This cost is typically well above the fraud
>> risk.
>>
>>>
>>
>>> It's important to recognize that bitcoin serves a wide variety of
>> use
>>
>>> cases with different profiles for time sensitivity and fraud risk.
>>
>>>
>>
>> I agree that unconfirmed transactions are incredibly important, but
>> not
>>
>> over SPV against random peers.
>>
>> If you offer users/merchants a feature (SPV 0-conf against random
>>
>> peers), that is fundamentally insecure, it will – sooner or later
>> – lead
>>
>> to some large scale fiasco, hurting Bitcoins reputation and trust
>> from
>>
>> merchants.
>>
>> Merchants using and trusting 0-conf SPV transactions (retrieved from
>>
>> random peers) is something we should **really eliminate** through
>>
>> education and by offering different solution.
>>
>> There are plenty, more sane options. If you can't run your own
>> full-node
>>
>> as a merchant (trivial), maybe co-use a wallet-service with
>> centralized
>>
>> verification (maybe use two of them), I guess Copay would be one of
>>
>> those wallets (as an example). Use them in watch-only mode.
>>
>> For end-users SPV software, I think it would be recommended to...
>>
>> ... disable unconfirmed transactions during SPV against random peers
>>
>> ... enable unconfirmed transactions when using SPV against a trusted
>>
>> peer with preshared keys after BIP150
>>
>> ... if unconfirmed transactions are disabled, show how it can be
>> enabled
>>
>> (how to run a full-node [in a box, etc.])
>>
>> ... educate, inform users that a transaction with no confirmation
>> can be
>>
>> "stopped" or "redirected" any time, also inform about the risks
>> during
>>
>> low-conf phase (1-5).
>>
>> I though see the point that it's nice to make use of the "incoming
>>
>> funds..." feature in SPV wallets. But – for the sake of stability
>> and
>>
>> (risk-)scaling – we may want to recommend to scarify this feature
>> and –
>>
>> in the same turn – to use privacy-preserving BFD's.
>>
>> </jonas>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-06 2:15 ` bfd
@ 2017-01-06 7:07 ` Aaron Voisine
0 siblings, 0 replies; 34+ messages in thread
From: Aaron Voisine @ 2017-01-06 7:07 UTC (permalink / raw)
To: bfd; +Cc: Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 5107 bytes --]
Credit card transactions are simply an expample of a widely used payment
system that has frequent fraud and chargebacks. The argument I'm making is
that different people in different situations value speed and convenience
over a known fraud risk. Instant zero confirmation transactions are
extremely useful despite the risk in all kinds of real world situations.
You can't substitute your own value judgements for everyone in every
situation.
Aaron
On Thu, Jan 5, 2017 at 6:15 PM <bfd@cock•lu> wrote:
> With a credit card you have an institution worth billions of dollars
>
> asserting that a payment has been made, with the option that it may be
>
> retracted under special circumstances by the card issuer.
>
>
>
> Unconfirmed Bitcoin transactions with a SPV client has you trusting
>
> that the un-authenticated DNS seed lookup has not been tampered with,
>
> the connection to the random node that you connect to has not been
>
> tampered with, and that the seed nor the node are attempting to
>
> manipulate you.
>
>
>
> The two scenarios aren’t even remotely comparable.
>
>
>
>
>
> On 2017-01-04 00:56, Aaron Voisine wrote:
>
> > It's easy enough to mark a transaction as "pending". People with bank
>
> > accounts are familiar with the concept.
>
> >
>
> > Although the risk of accepting gossip information from multiple random
>
> > peers, in the case where the sender does not control the receivers
>
> > network is still minimal. Random node operators have no incentive to
>
> > send fake transactions, and would need to control all the nodes a
>
> > client connects to, and find a non-false-positive address belonging to
>
> > the victims wallet.
>
> >
>
> > It's not impossible, but it's non trivial, would only temporarily show
>
> > a pending transaction, and provide no benefit to the node operator.
>
> > There are much juicier targets for an attacker with the ability to
>
> > sybil attack the entire bitcoin p2p network.
>
> >
>
> > Aaron
>
> >
>
> > On Tue, Jan 3, 2017 at 11:47 PM Jonas Schnelli <dev@jonasschnelli•ch>
>
> > wrote:
>
> >
>
> >> Hi
>
> >>
>
> >>> Unconfirmed transactions are incredibly important for real world
>
> >> use.
>
> >>
>
> >>> Merchants for instance are willing to accept credit card payments
>
> >> of
>
> >>
>
> >>> thousands of dollars and ship the goods despite the fact that the
>
> >>
>
> >>> transaction can be reversed up to 60 days later. There is a very
>
> >> large
>
> >>
>
> >>> cost to losing the ability to have instant transactions in many or
>
> >>
>
> >>> even most situations. This cost is typically well above the fraud
>
> >> risk.
>
> >>
>
> >>>
>
> >>
>
> >>> It's important to recognize that bitcoin serves a wide variety of
>
> >> use
>
> >>
>
> >>> cases with different profiles for time sensitivity and fraud risk.
>
> >>
>
> >>>
>
> >>
>
> >> I agree that unconfirmed transactions are incredibly important, but
>
> >> not
>
> >>
>
> >> over SPV against random peers.
>
> >>
>
> >> If you offer users/merchants a feature (SPV 0-conf against random
>
> >>
>
> >> peers), that is fundamentally insecure, it will – sooner or later
>
> >> – lead
>
> >>
>
> >> to some large scale fiasco, hurting Bitcoins reputation and trust
>
> >> from
>
> >>
>
> >> merchants.
>
> >>
>
> >> Merchants using and trusting 0-conf SPV transactions (retrieved from
>
> >>
>
> >> random peers) is something we should **really eliminate** through
>
> >>
>
> >> education and by offering different solution.
>
> >>
>
> >> There are plenty, more sane options. If you can't run your own
>
> >> full-node
>
> >>
>
> >> as a merchant (trivial), maybe co-use a wallet-service with
>
> >> centralized
>
> >>
>
> >> verification (maybe use two of them), I guess Copay would be one of
>
> >>
>
> >> those wallets (as an example). Use them in watch-only mode.
>
> >>
>
> >> For end-users SPV software, I think it would be recommended to...
>
> >>
>
> >> ... disable unconfirmed transactions during SPV against random peers
>
> >>
>
> >> ... enable unconfirmed transactions when using SPV against a trusted
>
> >>
>
> >> peer with preshared keys after BIP150
>
> >>
>
> >> ... if unconfirmed transactions are disabled, show how it can be
>
> >> enabled
>
> >>
>
> >> (how to run a full-node [in a box, etc.])
>
> >>
>
> >> ... educate, inform users that a transaction with no confirmation
>
> >> can be
>
> >>
>
> >> "stopped" or "redirected" any time, also inform about the risks
>
> >> during
>
> >>
>
> >> low-conf phase (1-5).
>
> >>
>
> >> I though see the point that it's nice to make use of the "incoming
>
> >>
>
> >> funds..." feature in SPV wallets. But – for the sake of stability
>
> >> and
>
> >>
>
> >> (risk-)scaling – we may want to recommend to scarify this feature
>
> >> and –
>
> >>
>
> >> in the same turn – to use privacy-preserving BFD's.
>
> >>
>
> >> </jonas>
>
>
[-- Attachment #2: Type: text/html, Size: 8494 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-05 7:45 ` Eric Voskuil
2017-01-05 14:48 ` Christian Decker
@ 2017-01-06 20:15 ` Chris Priest
2017-01-06 21:35 ` James MacWhyte
1 sibling, 1 reply; 34+ messages in thread
From: Chris Priest @ 2017-01-06 20:15 UTC (permalink / raw)
To: Eric Voskuil; +Cc: Bitcoin Protocol Discussion
Its a method for determining the probability that a valid tx will be
mined in a block before that tx actually gets mined, which is useful
when accepting payments in situations when you can't wait for the full
confirmation. No one is saying all tx validation should be performed
by querying miners mempools, that's ridiculous. Obviously once the tx
gets it's first confirmation, you go back to determining validity the
way you always have. There is no "security catastrophe".
Even if you're running a full node, you can't know for certain that
any given tx will make it into a future block. You can't be certain
the future miner who finally does mine that tx will mine your TXID or
another TXID that spends the same inputs to another address (a double
spend). The only way to actually know for certain is to query every
single large hashpower mempool.
On 1/4/17, Eric Voskuil <eric@voskuil•org> wrote:
> On 01/04/2017 11:06 PM, Chris Priest via bitcoin-dev wrote:
>> On 1/3/17, Jonas Schnelli via bitcoin-dev
>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>>
>>> There are plenty, more sane options. If you can't run your own full-node
>>> as a merchant (trivial), maybe co-use a wallet-service with centralized
>>> verification (maybe use two of them), I guess Copay would be one of
>>> those wallets (as an example). Use them in watch-only mode.
>>
>> The best way is to connect to the mempool of each miner and check to
>> see if they have your txid in their mempool.
>>
>> https://www.antpool.com/api/is_in_mempool?txid=334847bb...
>> https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
>> https://bw.com/api/is_in_mempool?txid=334847bb...
>> https://bitfury.com/api/is_in_mempool?txid=334847bb...
>> https://btcc.com/api/is_in_mempool?txid=334847bb...
>>
>> If each of these services return "True", and you know those services
>> so not engage in RBF, then you can assume with great confidence that
>> your transaction will be in the next block, or in a block very soon.
>> If any one of those services return "False", then you must assume that
>> it is possible that there is a double spend floating around, and that
>> you should wait to see if that tx gets confirmed. The problem is that
>> not every pool runs such a service to check the contents of their
>> mempool...
>>
>> This is an example of mining centralization increasing the security of
>> zero confirm.
>
> A world connected up to a few web services to determine payment validity
> is an example of a bitcoin security catastrophe.
>
> e
>
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-06 20:15 ` Chris Priest
@ 2017-01-06 21:35 ` James MacWhyte
2017-01-06 21:50 ` Eric Voskuil
0 siblings, 1 reply; 34+ messages in thread
From: James MacWhyte @ 2017-01-06 21:35 UTC (permalink / raw)
To: Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 3698 bytes --]
It's my opinion that the purpose of this list and bitcoin protocol
development in general is to build the base functionality that other
companies and individuals require to provide usability to the end-user. The
0-conf debate is a UX issue. If end users shouldn't rely on 0-conf, it is
up to wallet developers to hide 0-conf transactions or mark them
appropriately. Instead of using this list to debate what wallet designers
should or shouldn't do, we should just provide the tools and "let the
market sort it out". If wallet developers start getting inundated with
complaints that 0-conf transactions are causing confusion and loss, they
will find a solution. If the tools they require for the solution don't
exist, they will come to this list to request action.
Am I wrong?
On Fri, Jan 6, 2017 at 12:16 PM Chris Priest via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:
> Its a method for determining the probability that a valid tx will be
> mined in a block before that tx actually gets mined, which is useful
> when accepting payments in situations when you can't wait for the full
> confirmation. No one is saying all tx validation should be performed
> by querying miners mempools, that's ridiculous. Obviously once the tx
> gets it's first confirmation, you go back to determining validity the
> way you always have. There is no "security catastrophe".
>
> Even if you're running a full node, you can't know for certain that
> any given tx will make it into a future block. You can't be certain
> the future miner who finally does mine that tx will mine your TXID or
> another TXID that spends the same inputs to another address (a double
> spend). The only way to actually know for certain is to query every
> single large hashpower mempool.
>
> On 1/4/17, Eric Voskuil <eric@voskuil•org> wrote:
> > On 01/04/2017 11:06 PM, Chris Priest via bitcoin-dev wrote:
> >> On 1/3/17, Jonas Schnelli via bitcoin-dev
> >> <bitcoin-dev@lists•linuxfoundation.org> wrote:
> >>>
> >>> There are plenty, more sane options. If you can't run your own
> full-node
> >>> as a merchant (trivial), maybe co-use a wallet-service with centralized
> >>> verification (maybe use two of them), I guess Copay would be one of
> >>> those wallets (as an example). Use them in watch-only mode.
> >>
> >> The best way is to connect to the mempool of each miner and check to
> >> see if they have your txid in their mempool.
> >>
> >> https://www.antpool.com/api/is_in_mempool?txid=334847bb...
> >> https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
> >> https://bw.com/api/is_in_mempool?txid=334847bb...
> >> https://bitfury.com/api/is_in_mempool?txid=334847bb...
> >> https://btcc.com/api/is_in_mempool?txid=334847bb...
> >>
> >> If each of these services return "True", and you know those services
> >> so not engage in RBF, then you can assume with great confidence that
> >> your transaction will be in the next block, or in a block very soon.
> >> If any one of those services return "False", then you must assume that
> >> it is possible that there is a double spend floating around, and that
> >> you should wait to see if that tx gets confirmed. The problem is that
> >> not every pool runs such a service to check the contents of their
> >> mempool...
> >>
> >> This is an example of mining centralization increasing the security of
> >> zero confirm.
> >
> > A world connected up to a few web services to determine payment validity
> > is an example of a bitcoin security catastrophe.
> >
> > e
> >
> >
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
[-- Attachment #2: Type: text/html, Size: 6443 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-06 21:35 ` James MacWhyte
@ 2017-01-06 21:50 ` Eric Voskuil
0 siblings, 0 replies; 34+ messages in thread
From: Eric Voskuil @ 2017-01-06 21:50 UTC (permalink / raw)
To: James MacWhyte, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 4773 bytes --]
It is a useful aspect of discussion at this level as it helps higher lever developers understand the actual tradeoffs. Clearly some do not. The market will eventually sort them out, but the discussion both gives developers the necessary information.
It also helps core development prioritize resources. I personally would not prioritize core work to facilitate zero conf. I would even spend time to discourage it, as others have done.
I think the cautions in this thread about doing privacy and system security damaging things (like checking mining pools for zero conf transactions) will prevent some wasted time, which benefits everyone.
e
> On Jan 6, 2017, at 1:35 PM, James MacWhyte via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
>
> It's my opinion that the purpose of this list and bitcoin protocol development in general is to build the base functionality that other companies and individuals require to provide usability to the end-user. The 0-conf debate is a UX issue. If end users shouldn't rely on 0-conf, it is up to wallet developers to hide 0-conf transactions or mark them appropriately. Instead of using this list to debate what wallet designers should or shouldn't do, we should just provide the tools and "let the market sort it out". If wallet developers start getting inundated with complaints that 0-conf transactions are causing confusion and loss, they will find a solution. If the tools they require for the solution don't exist, they will come to this list to request action.
>
> Am I wrong?
>
> On Fri, Jan 6, 2017 at 12:16 PM Chris Priest via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
>> Its a method for determining the probability that a valid tx will be
>> mined in a block before that tx actually gets mined, which is useful
>> when accepting payments in situations when you can't wait for the full
>> confirmation. No one is saying all tx validation should be performed
>> by querying miners mempools, that's ridiculous. Obviously once the tx
>> gets it's first confirmation, you go back to determining validity the
>> way you always have. There is no "security catastrophe".
>>
>> Even if you're running a full node, you can't know for certain that
>> any given tx will make it into a future block. You can't be certain
>> the future miner who finally does mine that tx will mine your TXID or
>> another TXID that spends the same inputs to another address (a double
>> spend). The only way to actually know for certain is to query every
>> single large hashpower mempool.
>>
>> On 1/4/17, Eric Voskuil <eric@voskuil•org> wrote:
>> > On 01/04/2017 11:06 PM, Chris Priest via bitcoin-dev wrote:
>> >> On 1/3/17, Jonas Schnelli via bitcoin-dev
>> >> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>> >>>
>> >>> There are plenty, more sane options. If you can't run your own full-node
>> >>> as a merchant (trivial), maybe co-use a wallet-service with centralized
>> >>> verification (maybe use two of them), I guess Copay would be one of
>> >>> those wallets (as an example). Use them in watch-only mode.
>> >>
>> >> The best way is to connect to the mempool of each miner and check to
>> >> see if they have your txid in their mempool.
>> >>
>> >> https://www.antpool.com/api/is_in_mempool?txid=334847bb...
>> >> https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
>> >> https://bw.com/api/is_in_mempool?txid=334847bb...
>> >> https://bitfury.com/api/is_in_mempool?txid=334847bb...
>> >> https://btcc.com/api/is_in_mempool?txid=334847bb...
>> >>
>> >> If each of these services return "True", and you know those services
>> >> so not engage in RBF, then you can assume with great confidence that
>> >> your transaction will be in the next block, or in a block very soon.
>> >> If any one of those services return "False", then you must assume that
>> >> it is possible that there is a double spend floating around, and that
>> >> you should wait to see if that tx gets confirmed. The problem is that
>> >> not every pool runs such a service to check the contents of their
>> >> mempool...
>> >>
>> >> This is an example of mining centralization increasing the security of
>> >> zero confirm.
>> >
>> > A world connected up to a few web services to determine payment validity
>> > is an example of a bitcoin security catastrophe.
>> >
>> > e
>> >
>> >
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[-- Attachment #2: Type: text/html, Size: 7876 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2016-07-28 21:07 ` Leo Wandersleb
@ 2017-01-06 22:07 ` Erik Aronesty
0 siblings, 0 replies; 34+ messages in thread
From: Erik Aronesty @ 2017-01-06 22:07 UTC (permalink / raw)
To: Leo Wandersleb, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 4559 bytes --]
- N \log_2 \epsilon * 1.44
N = 41000 blocks
epsilon = 1/41000 (fp rate)
= 904689.8bits
~ 1 MB
On Thu, Jul 28, 2016 at 5:07 PM, Leo Wandersleb via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:
> gmaxwell just made me aware of this mail thread [0]. Some days ago I had
> independently and naively started implementing "something similar" [1].
>
> My version totally ignored the commitment and signing part but I'm pretty
> sure
> that 12GB is overkill. My code is currently broken and I have no time to
> work on
> it much but I thought it might be helpful to chime in.
>
> At this point in time the difference between 80GB and 3GB (as my current
> 1.5GB
> of only outputs would suggest if I had covered the inputs) or even 12GB
> makes
> the difference of being able to store it on a phone, vs. not being able
> to. 80GB
> "compressed" to 3GB is not that bad at all. Unfortunately, with segWit
> this will
> be worse, with the higher transaction count per MB.
>
> Regards,
>
> Leo
>
> [0]
> https://www.reddit.com/r/Bitcoin/comments/4v28jl/how_
> have_fungiblity_problems_affected_you_in/d5ux6aq
> [1] https://github.com/Giszmo/TransactionFinder
>
> On 05/11/2016 10:29 PM, Bob McElrath via bitcoin-dev wrote:
> > Eerrrr....let me revise that last paragraph. That's 12 *GB* of filters
> at
> > today's block height (at fixed false-positive rate 1e-6. Compared to
> block
> > headers only which are about 33 MB today. So this proposal is not really
> > compatible with such a wallet being "light"...
> >
> > Damn units...
> >
> > Bob McElrath via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org]
> wrote:
> >> I like this idea, but let's run some numbers...
> >>
> >> bfd--- via bitcoin-dev [bitcoin-dev@lists•linuxfoundation.org] wrote:
> >>> A Bloom Filter Digest is deterministically created of every block
> >> Bloom filters completely obfuscate the required size of the filter for
> a desired
> >> false-positive rate. But, an optimal filter is linear in the number of
> elements
> >> it contains for fixed false-positive rate, and logarithmic in the
> false-positive
> >> rate. (This comment applies to a RLL encoded Bloom filter Greg
> mentioned, but
> >> that's not the only way) That is for N elements and false positive rate
> >> \epsilon:
> >>
> >> filter size = - N \log_2 \epsilon
> >>
> >> Given that the data that would be put into this particular filter is
> *already*
> >> hashed, it makes more sense and is faster to use a Cuckoo[1] filter,
> choosing a
> >> fixed false-positive rate, given expected wallet sizes. For Bloom
> filters,
> >> multiply the above formula by 1.44.
> >>
> >> To prevent light clients from downloading more blocks than necessary,
> the
> >> false-positive rate should be roughly less than 1/(block height). If
> we take
> >> the false positive rate to be 1e-6 for today's block height ~ 410000,
> this is
> >> about 20 bits per element. So for todays block's, this is a 30kb
> filter, for a
> >> 3% increase in block size, if blocks commit to the filter. Thus the
> required
> >> size of the filter commitment is roughly:
> >>
> >> filter size = N \log_2 H
> >>
> >> where H is the block height. If bitcoin had these filters from the
> beginning, a
> >> light client today would have to download about 12MB of data in
> filters. My
> >> personal SPV wallet is using 31MB currently. It's not clear this is a
> bandwidth
> >> win, though it's definitely a win for computing load on full nodes.
> >>
> >>
> >> [1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
> >>
> >> --
> >> Cheers, Bob McElrath
> >>
> >> "For every complex problem, there is a solution that is simple, neat,
> and wrong."
> >> -- H. L. Mencken
> >>
> >>
> >>
> >> !DSPAM:5733934b206851108912031!
> >
> >
> >> _______________________________________________
> >> bitcoin-dev mailing list
> >> bitcoin-dev@lists•linuxfoundation.org
> >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> >>
> >>
> >> !DSPAM:5733934b206851108912031!
> > --
> > Cheers, Bob McElrath
> >
> > "For every complex problem, there is a solution that is simple, neat,
> and wrong."
> > -- H. L. Mencken
> >
> >
> >
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists•linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
[-- Attachment #2: Type: text/html, Size: 6811 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2016-05-09 8:26 [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security bfd
` (2 preceding siblings ...)
[not found] ` <77b6dd25-0603-a0bd-6a9e-38098e5cb19d@jonasschnelli.ch>
@ 2017-02-17 0:28 ` Chris Belcher
2017-04-01 23:49 ` bfd
3 siblings, 1 reply; 34+ messages in thread
From: Chris Belcher @ 2017-02-17 0:28 UTC (permalink / raw)
To: bitcoin-dev
I believe this proposal still suffers from one problem that bip37 did,
albiet by a much lesser extent. Combining the partial information from
the block downloads with the transaction subgraph information from the
blockchain can in some cases still reveal which addresses belong to the
wallet. Nonetheless this proposal still has many benefits and is well
worth working on.
==BIP37==
As a recap, probably the biggest and most problematic way that bip37 was
broken was by combining the partial wallet information from the bloom
filter with the transaction subgraph information from the blockchain
Suppose a wallet synchronizes it's history, if it spent a coin from its
address A, it must also also add the change address B to the bloom
filter, which is connected to A directly on transaction graph.
As an example, consider five typical transactions that consume one input
each and produce two outputs.
A, B, C, D, E refer to transactions. A1, A2, etc refer to addresses
within those transactions
-> C1
A1 -> B2 -> C2
-> B2 -> D1
-> D2 -> E1
-> E2
If a bip37 bloom filter matches addresses A1, B2, D2, E1 then it can be
seen that they form a "peel chain" [this terminology comes from
https://cseweb.ucsd.edu/~smeiklejohn/files/imc13.pdf]
-> X
A1 -> X -> X
-> B2 -> X
-> D2 -> E1
-> X
The same five transactions with non-matching addresses replaced by X.
The peel chain is visible, it's clear that B2, D2, E1 are change
addresses which belong to the same wallet as A1.
For a given false-positive rate fp and given length of peel chain C, the
odds of a false positive peel chain happening by chance is fp^C which
rapidly gets very small as the wallet makes more transactions (increases C).
If only one address was matched from the above group (for example B2)
then it likely to be a false positive by the fact that it doesn't make
any transactions to another address that also matches the bloom filter.
Another possibility is that the address is a payment output that the
wallet received but hasn't spent yet, but the wallet cant spend it
without adding the change address to the bloom filter and thus revealing
itself to the spy.
I believe the committed bloom filter proposal is vulnerability to this
same kind of attack because it still leaks information about which
addresses the wallet is interested in.
==Committed Bloom Filter Maths==
I'll try to analyze this now. I'll find the expectation value of the
number of transaction subgraphs in those blocks that appear just by
chance. If this expectation goes to zero, then the only transaction
subgraph left will be the real one that the wallet is actually
interested in. In that case it will be possible to spy on the wallet.
Assuming outputs have the same probability of being spent in each time
interval (i.e. they are spent in a Poisson process) This is
approximately true, see the graphs from
[https://letstalkbitcoin.com/blog/post/rise-of-the-zombie-bitcoins].
This means we can assign
a single probability P that an output is spent in each block.
Assume every transaction has one change address only and spending of
unconfirmed change doesn't happen (its more efficient to use RBF to add
additional outputs anyway)
Number of transactions per block = Q (about 1800 today)
Number of outputs per block = Z = 2*Q (approximately)
Length of peel chain = Number of transactions in wallet = C
Average time an output is unspent for = T (about 1 month, very roughly
estimating from the above blog post)
Probability an output being spent in any particular later block = P =
10minutes/T
Assume no false positive blocks
Say wallet downloaded two blocks and they are ordered by block height
The expected number of tx subgraphs between them, E(#G)
E(#G) = number of outputs created in block 1 that get spent in block 2
= Z*P
Say the wallet downloaded three blocks
Expected number of subgraphs going through them all
E(#G) = number of outputs created in block 1 get spent in block 2, that
create a change address which gets spent in block 3
= Z*P*P
Say the wallet downloaded C blocks
Expected number of tx subgraphs going through all the blocks by chance
E(#G) = Z*P^C
which gets small quickly as C goes up, because P < 1
Now drop the assumption about no false positive blocks.
Let the number of candidate blocks be D.
This is how many blocks the wallet scans, it's related to how far in the
past the wallet's keys was created. At one extreme wallet was created at
genesis block and so D = ~450000, at other extreme created now so D = 0.
Note that D = 0 must also imply C = 0
Expected number of false positive blocks downloaded = F = fp*D
In all these situations the blocks are sorted by block height
Suppose have C=2, F=1, and false one is in the middle.
I want to find E(#G|CF), the expected number of transaction subgraphs
that appear just by chance, given C and F.
E(#G|CF) = how many outputs which are created in block 1 get spent in
block 3
= Z*P
Same situation, but false one at the start instead of middle.
E(#G|CF) = how many outputs which are created in block 2 get spent in
block 3
= Z*P
Same situation but false one could be anywhere, result in the sum of the
probability for any false block position
E(#G|CF) = C(3, 1)*Z*P = 3*Z*P
where C() is the number of order-independent ways of choosing 1 element
out of a set of 3 elements, also known as the binomial coefficient
Now suppose C=3 and F=1
The same argument leads to
E(#G|CF) = C(4, 1)*Z*P^2 = 4*Z*P^2
Now suppose C=3 and F=2, with fp blocks at the end
E(#G|CF)
= how many outputs are created in block 1, are spent in block 2 and
change address spent in block 3
= Z*P^2
Same situation but fp blocks can be anywhere, add up all the possible
combinations of them within the rest
E(#G|CF) = C(5, 2)*Z*P^2 = 5*Z*P^2
With these same rules, its clear the general expression for any F and C
E(#G|CF) = C(F + C, F)*Z*P^(C - 1)
A more interesting value might be the time evolution of E(#G)
Let B be the blocks in the blockchain since the wallet creation date, as you
know it increases at an average rate of one every ten minutes
w = wallet transaction creation rate, expressed per-block
C = w * B
F = fp * B
J = average blocks between wallet transactions = 1440 (10 days)
w = 1/J
E(#G|B) = C((fp + w)*B, fp*B)*Z*P^(w*B - 1)
This goes to zero as B becomes big, although choosing very high values
of fp makes it go to zero slower.
This is only approximate maths, in actuality you cannot take the number
of false positive blocks to be fp*B, you have to sum over all blocks
weighted by probability. And outputs might not be spent in an exact
Poisson process so you cant just multiply by P each time. Plus if your
false positive rate is very high then some of your false positive blocks
will actually contain your real transactions, this analysis
double-counts them.
Using some reasonable values and plotting E(#G|B) against B can show how
quickly it drops and therefore leaves only the true transaction subgraph.
(note: in LibreOffice Math and Microsoft Excel the binomial coefficient
function is COMBIN)
==Notes==
*) The expected number of transaction subgraphs that happen by chance
goes to zero eventually as the blockchain steps ahead. Unless the fp
rate is very high (close to 1) and time between wallet transaction very
long, in which case the binomial coefficient term gets larger more
quicker than the exponential decay P^B term gets smaller.
*) fp rate doesn't help in most cases that much compared to the
exponential drop-off from time ticking ahead requiring more downloading
of blocks
*) its good for privacy if bitcoin outputs are spent more frequently so
P is higher, because that creates more transaction subgraphs in the
anonymity set.
*) its good for privacy if more outputs are made per block, although
still only linearly which is no match for the exponential reduction from
the P^B term.
*) its good for privacy to make less of your own transactions (increase
J and reduce w), for low-activity users the privacy of committed bloom
filters can be actually pretty good, for high-activity users who use
bitcoin's blockchain all the time it's not very good
*) For the reasonable values I tried for a once-a-month user with fp=1%,
their chance-transaction-subgraph-count drops below 1 in about eight months.
*) Because of the exponential nature, E(#G) goes from "billions of
billions" to "about 10" fairly quickly.
==Discussion of ways to mitigate this==
One way is to not use change outputs. This is unrealistic, doesn't match
people's behavior and money must be divisible.
A better way to mitigate this is to not leak the information that all
those blocks are interesting to the same wallet. Don't download all
blocks from the same archival node. If you download blocks from many
different nodes, it gives an incentive for surveillance startups to
create lots of sybil nodes they control and can then correlate together
block downloads with the wallet IP address. Many such startups are
already doing this today to try to detect the origin IP address of
broadcasted transactions.
Another solution could be to download a few blocks from different nodes
with new tor circuits used. This would delink the wallet IP address from
the downloads and would help a lot. This has the issue that tor is
slower (but still not as slow as downloading the entire blockchain)
Another way a wallet could be correlated with its block downloads is
timing correlations. At any one time only a certain number of peers
would be downloading blocks which narrows down which wallets are
downloading what. However even today Bitcoin Core downloads blocks in
parallel from many nodes so there's probably quite a large anonymity set
for lightweight wallets using committed bloom filters. Plus timing
correlation can be reduced simply by waiting longer. Wallets are not
sync'd from backup very often so it might be okay to wait.
Another way to improve privacy could be for the wallet to choose random
transaction subgraphs and download all the blocks related to them as well.
Wallet developers might choose to allow the user to configure their own
fp rate. This is probably not a good idea since the relationship between
fp rate and anonymity set is non-obvious. It might be better to ask the
user how often they expect to make transactions.
==Conclusion==
I think this committed bloom filter idea is very good and much better
than bip37, but for good privacy for when bitcoin is used often still
requires certain behavior namely downloading blocks
from many different peers with new tor circuits.
Note that I've been dealing with counting transaction subgraphs but
actually finding them from blocks might also be computationally
infeasible. Although a Bayesian approach worked very
well for similar transaction subgraph linking
[https://arxiv.org/pdf/1612.06747v3.pdf]
It would also be interesting to analyze what information a spy can get
if they are missing some blocks that the wallet downloaded.
For the long term, private and high-volume bitcoin use will be best
served by off-chain transactions. They will probably be a huge win just
because the large and public blockchain is such a non-private
way of doing things.
On 09/05/16 09:26, bfd--- via bitcoin-dev wrote:
> We introduce several concepts that rework the lightweight Bitcoin
> client model in a manner which is secure, efficient and privacy
> compatible.
>
> Thea properties of BIP37 SPV [0] are unfortunately not as strong as
> originally thought:
>
> * The expected privacy of the probabilistic nature of bloom
> filters does not exist [1][2], any user with a BIP37 SPV wallet
> should be operating under no expectation of privacy.
> Implementation flaws make this effect significantly worse, the
> behavior meaning that no matter how high the false positive
> rate (up to simply downloading the whole blocks verbatim) the
> intent of the client connection is recoverable.
>
> * Significant processing load is placed on nodes in the Bitcoin
> network by lightweight clients, a single syncing wallet causes
> (at the time of writing) 80GB of disk reads and a large amount
> of CPU time to be consumed processing this data. This carries
> significant denial of service risk [3], non-distinguishable
> clients can repeatedly request taxing blocks causing
> reprocessing on every request. Processed data is unique to every
> client, and can not be cached or made more efficient while
> staying within specification.
>
> * Wallet clients can not have strong consistency or security
> expectations, BIP37 merkle paths allow for a wallet to validate
> that an output was spendable at some point in time but does not
> prove that this output is not spent today.
>
> * Nodes in the network can denial of service attack all BIP37 SPV
> wallet clients by simply returning null filter results for
> requests, the wallet has no way of discerning if it has been
> lied to and may be made simply unaware that any payment has been
> made to them. Many nodes can be queried in a probabilistic manor
> but this increases the already heavy network load with little
> benefit.
>
>
>
> We propose a new concept which can work towards addressing these
> shortcomings.
>
>
> A Bloom Filter Digest is deterministically created of every block
> encompassing the inputs and outputs of the containing transactions,
> the filter parameters being tuned such that the filter is a small
> portion of the size of the total block data. To determine if a block
> has contents which may be interesting a second bloom filter of all
> relevant key material is created. A binary comparison between the two
> filters returns true if there is probably matching transactions, and
> false if there is certainly no matching transactions. Any matched
> blocks can be downloaded in full and processed for transactions which
> may be relevant.
>
> The BFD can be used verbatim in replacement of BIP37, where the filter
> can be cached between clients without needing to be recomputed. It can
> also be used by normal pruned nodes to do re-scans locally of their
> wallet without needing to have the block data available to scan, or
> without reading the entire block chain from disk.
>
> -
>
> For improved probabilistic security the bloom filters can be presented
> to lightweight clients by semi-trusted oracles. A client wallet makes
> an assumption that they trust a set, or subset of remote parties
> (wallet vendors, services) which all all sign the BFD for each block.
> The BFD can be downloaded from a single remote source, and the hash of
> the filters compared against others in the trust set. Agreement is a
> weak suggestion that the filter has not been tampered with, assuming
> that these parties are not conspiring to defraud the client.
>
> The oracles do not learn any additional information about the client
> wallet, the client can download the block data from either nodes on
> the network, HTTP services, NTTP, or any other out of band
> communication method that provides the privacy desired by the client.
>
> -
>
> The security model of the oracle bloom filter can be vastly improved
> by instead committing a hash of the BFD inside every block as a soft-
> fork consensus rule change. After this, every node in the network would
> build the filter and validate that the hash in the block is correct,
> then make a conscious choice discard it for space savings or cache the
> data to disk.
>
> With a commitment to the filter it becomes impossible to lie to
> lightweight clients by omission. Lightweight clients are provided with
> a block header, merkle path, and the BFD. Altering the BFD invalidates
> the merkle proof, it's validity is a strong indicator that the client
> has an unadulterated picture of the UTXO condition without needing to
> build one itself. A strong assurance that the hash of the BFD means
> that the filters can be downloaded out of band along with the block
> data at the leisure of the client, allowing for significantly greater
> privacy and taking load away from the P2P Bitcoin network.
>
> Committing the BFD is not a hard forking change, and does not require
> alterations to mining software so long as the coinbase transaction
> scriptSig is not included in the bloom filter.
>
>
> [0] https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki
> [1] https://eprint.iacr.org/2014/763.pdf
> [2] https://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj/
> [3] https://github.com/petertodd/bloom-io-attack
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-01-06 2:04 ` bfd
@ 2017-03-15 22:36 ` Tom Harding
2017-03-16 0:25 ` bfd
0 siblings, 1 reply; 34+ messages in thread
From: Tom Harding @ 2017-03-15 22:36 UTC (permalink / raw)
To: bitcoin-dev
Agreed.
In contrast, BIP37 as used today is totally decentralized, and can me
made much more secure, private, and scalable -- without giving up the
utility of unconfirmed transactions.
Please don't read into this statement a belief that all the coffees
should go on the chain, or that the security or privacy of BIP37 compare
favorably to any other particular thing.
https://docs.google.com/presentation/d/13MzUo2iIH9JBW29TgtPMoaMXxeEdanWDfi6SlfO-LlA
On 1/5/2017 6:04 PM, bfd--- via bitcoin-dev wrote:
> You might as well replace Bitcoin with a system where these parties
> sign transactions and skip mining altogether, it would have the same
> properties and be significantly more effient.
>
>
> On 2017-01-04 23:06, Chris Priest wrote:
>> On 1/3/17, Jonas Schnelli via bitcoin-dev
>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>>
>>> There are plenty, more sane options. If you can't run your own
>>> full-node
>>> as a merchant (trivial), maybe co-use a wallet-service with centralized
>>> verification (maybe use two of them), I guess Copay would be one of
>>> those wallets (as an example). Use them in watch-only mode.
>>
>> The best way is to connect to the mempool of each miner and check to
>> see if they have your txid in their mempool.
>>
>> https://www.antpool.com/api/is_in_mempool?txid=334847bb...
>> https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
>> https://bw.com/api/is_in_mempool?txid=334847bb...
>> https://bitfury.com/api/is_in_mempool?txid=334847bb...
>> https://btcc.com/api/is_in_mempool?txid=334847bb...
>>
>> If each of these services return "True", and you know those services
>> so not engage in RBF, then you can assume with great confidence that
>> your transaction will be in the next block, or in a block very soon.
>> If any one of those services return "False", then you must assume that
>> it is possible that there is a double spend floating around, and that
>> you should wait to see if that tx gets confirmed. The problem is that
>> not every pool runs such a service to check the contents of their
>> mempool...
>>
>> This is an example of mining centralization increasing the security of
>> zero confirm. If more people mined, this method will not work as well
>> because it would require you to call the API of hundreds of different
>> potential block creators.
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-03-15 22:36 ` Tom Harding
@ 2017-03-16 0:25 ` bfd
2017-03-16 15:05 ` Tom Harding
0 siblings, 1 reply; 34+ messages in thread
From: bfd @ 2017-03-16 0:25 UTC (permalink / raw)
To: Tom Harding, Bitcoin Protocol Discussion
Sorry, this is not the case.
Your slides gloss over the simple fact that compact fraud proofs in
Bitcoin aren't possible, and that the "SPV" implemented today bears
absolutely no resemblance in security properties to the version
described in the Bitcoin white paper. In the white paper SPV clients
have the same security as fully validating nodes, in the implementation
of BIP37 they have absolutely no security except the vague hope that
they are not being lied to, and that the chain with the most work they
are seeing is actually valid, both are very weak assumptions.
The suggested solution in no way precludes unconfirmed transactions from
being used with a commitment scheme, my comments and others are just
recognising that they are an almost useless indicator for people who
aren't validating.
During the validationless mining failure around the BIP66 activation
miners produced 6 invalid blocks in a chain, and many more invalid
blocks in isolated bursts for a period lasting several months. Due to
the instability of the network you are completely unreasonable to accept
anything except multiple confirmations, the true number for safety is
probably more like 60 or 120, not 6, or 3, or 1 as some Bitcoin
exchanges use today.
0000000000000000009cc829aa25b40b2cd4eb83dd498c12ad0d26d90c439d99.bin
bad-version(0x00000002)
0000000000000000032527aa796d3672e32e5f85a452d3a584a28fc7efbcd5d0.bin
bad-version(0x00000002)
000000000000000003ae1223f4926ec86100885cfe1484dc52fd67e042a19b12.bin
bad-version(0x00000002)
0000000000000000083cbdbb25c1607527c8f3fdb16f0d048c4439a73b501cb6.bin
bad-version(0x00000002)
00000000000000000954ed93eda1e79e8261137548fa9ccf4d516bb384a3660b.bin
bad-version(0x00000002)
00000000000000000afc9fbe7cfe8a6b50502d509ba626beb2e2d6c15d1d3ee3.bin
bad-version(0x00000002)
00000000000000000b6adf92bc192b3c21210f456ab21b5e46951665c74cfab2.bin
bad-version(0x00000002)
00000000000000000c9bb4a508fff34f5450d9c62ef2cb833e53909a4c549de5.bin
bad-version(0x00000002)
0000000000000000116322b5f25826787b01f7a70fb322837b68dff8216cefc4.bin
bad-version(0x00000002)
000000000000000012aac0664cd8b6cbc3ea485921a05f2c4340f928b0226d3c.bin
bad-version(0x00000002)
"SPV" like you're describing can exist, or validationless mining can
exist, both can not simultaneously.
On 2017-03-16 09:36, Tom Harding via bitcoin-dev wrote:
> Agreed.
>
> In contrast, BIP37 as used today is totally decentralized, and can me
> made much more secure, private, and scalable -- without giving up the
> utility of unconfirmed transactions.
>
> Please don't read into this statement a belief that all the coffees
> should go on the chain, or that the security or privacy of BIP37
> compare favorably to any other particular thing.
>
> https://docs.google.com/presentation/d/13MzUo2iIH9JBW29TgtPMoaMXxeEdanWDfi6SlfO-LlA
>
>
>
> On 1/5/2017 6:04 PM, bfd--- via bitcoin-dev wrote:
>> You might as well replace Bitcoin with a system where these parties
>> sign transactions and skip mining altogether, it would have the same
>> properties and be significantly more effient.
>>
>>
>> On 2017-01-04 23:06, Chris Priest wrote:
>>> On 1/3/17, Jonas Schnelli via bitcoin-dev
>>> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>>>>
>>>> There are plenty, more sane options. If you can't run your own
>>>> full-node
>>>> as a merchant (trivial), maybe co-use a wallet-service with
>>>> centralized
>>>> verification (maybe use two of them), I guess Copay would be one of
>>>> those wallets (as an example). Use them in watch-only mode.
>>>
>>> The best way is to connect to the mempool of each miner and check to
>>> see if they have your txid in their mempool.
>>>
>>> https://www.antpool.com/api/is_in_mempool?txid=334847bb...
>>> https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
>>> https://bw.com/api/is_in_mempool?txid=334847bb...
>>> https://bitfury.com/api/is_in_mempool?txid=334847bb...
>>> https://btcc.com/api/is_in_mempool?txid=334847bb...
>>>
>>> If each of these services return "True", and you know those services
>>> so not engage in RBF, then you can assume with great confidence that
>>> your transaction will be in the next block, or in a block very soon.
>>> If any one of those services return "False", then you must assume
>>> that
>>> it is possible that there is a double spend floating around, and that
>>> you should wait to see if that tx gets confirmed. The problem is that
>>> not every pool runs such a service to check the contents of their
>>> mempool...
>>>
>>> This is an example of mining centralization increasing the security
>>> of
>>> zero confirm. If more people mined, this method will not work as well
>>> because it would require you to call the API of hundreds of different
>>> potential block creators.
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-03-16 0:25 ` bfd
@ 2017-03-16 15:05 ` Tom Harding
0 siblings, 0 replies; 34+ messages in thread
From: Tom Harding @ 2017-03-16 15:05 UTC (permalink / raw)
To: bfd, Bitcoin Protocol Discussion
On 3/15/2017 5:25 PM, bfd@cock•lu wrote:
> compact fraud proofs in Bitcoin aren't possible
> In the white paper SPV clients have the same security as fully
> validating nodes
In addition to not existing, if compact fraud proofs did exist, trying
to ensure they are seen by SPV clients has the same problems as BIP37.
> in the implementation of BIP37 they have absolutely no security except
> the vague hope that they are not being lied to, and that the chain
> with the most work they are seeing is actually valid, both are very
> weak assumptions.
Since real money is involved, the near total absence of documented fraud
along these lines belies the strong language.
> During the validationless mining failure around the BIP66 activation
> miners produced 6 invalid blocks in a chain, and many more invalid
> blocks in isolated bursts for a period lasting several months. Due to
> the instability of the network you are completely unreasonable to
> accept anything except multiple confirmations
This affected all users, not just SPV.
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
2017-02-17 0:28 ` Chris Belcher
@ 2017-04-01 23:49 ` bfd
0 siblings, 0 replies; 34+ messages in thread
From: bfd @ 2017-04-01 23:49 UTC (permalink / raw)
To: Chris Belcher, Bitcoin Protocol Discussion
On 2017-02-17 11:28, Chris Belcher via bitcoin-dev wrote:
> I think this committed bloom filter idea is very good and much better
> than bip37, but for good privacy for when bitcoin is used often still
> requires certain behavior namely downloading blocks
> from many different peers with new tor circuits.
>
> Note that I've been dealing with counting transaction subgraphs but
> actually finding them from blocks might also be computationally
> infeasible. Although a Bayesian approach worked very
> well for similar transaction subgraph linking
> [https://arxiv.org/pdf/1612.06747v3.pdf]
>
> It would also be interesting to analyze what information a spy can get
> if they are missing some blocks that the wallet downloaded.
>
> For the long term, private and high-volume bitcoin use will be best
> served by off-chain transactions. They will probably be a huge win just
> because the large and public blockchain is such a non-private
> way of doing things.
>
Thank you for the analysis, this generally matches my views about the
properties offered by the system.
I've generally developed the opinion that BIP37 is effectively unused
by all but a very small number of wallets and services now, setting up
sinkhole nodes in the network to monitor `filterload` commands seems
to back that up.
^ permalink raw reply [flat|nested] 34+ messages in thread
end of thread, other threads:[~2017-04-01 23:49 UTC | newest]
Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-05-09 8:26 [bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security bfd
2016-05-09 8:57 ` Gregory Maxwell
2016-05-11 20:06 ` Bob McElrath
2016-05-11 20:29 ` Bob McElrath
2016-07-28 21:07 ` Leo Wandersleb
2017-01-06 22:07 ` Erik Aronesty
2017-01-03 20:24 ` bfd
[not found] ` <77b6dd25-0603-a0bd-6a9e-38098e5cb19d@jonasschnelli.ch>
2017-01-03 20:18 ` bfd
2017-01-03 22:18 ` Aaron Voisine
2017-01-03 22:28 ` bfd
2017-01-03 23:06 ` adiabat
2017-01-03 23:46 ` Aaron Voisine
2017-01-04 0:10 ` bfd
2017-01-04 0:36 ` Aaron Voisine
2017-01-04 6:06 ` Eric Voskuil
2017-01-04 16:13 ` Leo Wandersleb
2017-01-04 7:47 ` Jonas Schnelli
2017-01-04 8:56 ` Aaron Voisine
2017-01-04 10:13 ` Jorge Timón
2017-01-04 11:00 ` Adam Back
2017-01-06 2:15 ` bfd
2017-01-06 7:07 ` Aaron Voisine
2017-01-05 7:06 ` Chris Priest
2017-01-05 7:45 ` Eric Voskuil
2017-01-05 14:48 ` Christian Decker
2017-01-06 20:15 ` Chris Priest
2017-01-06 21:35 ` James MacWhyte
2017-01-06 21:50 ` Eric Voskuil
2017-01-06 2:04 ` bfd
2017-03-15 22:36 ` Tom Harding
2017-03-16 0:25 ` bfd
2017-03-16 15:05 ` Tom Harding
2017-02-17 0:28 ` Chris Belcher
2017-04-01 23:49 ` bfd
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox