* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-21 22:56 ` Andreas Schildbach
@ 2019-07-22 5:01 ` Matt Corallo
2019-07-22 15:58 ` Justus Ranvier
2019-07-22 8:32 ` Peter Todd
` (3 subsequent siblings)
4 siblings, 1 reply; 23+ messages in thread
From: Matt Corallo @ 2019-07-22 5:01 UTC (permalink / raw)
To: Andreas Schildbach, Bitcoin Protocol Discussion
Hey Andreas,
I think maybe some of the comments here were misunderstood - I don't
anticipate that most people will change their defaults, indeed, but
given the general upgrade cycles we've seen on the network over the
entire course of Bitcoin's history, there's little reason to believe
that many nodes with NODE_BLOOM publicly accessible will be around for
at least three or four years to come, though obviously any conscious
effort by folks who need those services to run nodes could extend that
significantly.
As for the DoS issues, a super old Proof-of-Concept of the I/O variant
is here: https://github.com/petertodd/bloom-io-attack though CPU DoS
attacks are also possible that use high hash counts to fill a node's CPU
usage (you can pretty trivially see when a bloom-based peer connects to
you just by looking at top...).
Finally, regarding alternatives, the filter-generation code for BIP
157/158 has been in Bitcoin Core for some time, though the P2P serving
side of things appears to have lost any champions working on it. I
presume one of the Lightning folks will eventually, given they appear to
be requiring their users connect to a handful of their own servers right
now, but if you really need it, its likely not a ton of work to pipe
them through.
Matt
On 7/21/19 10:56 PM, Andreas Schildbach via bitcoin-dev wrote:
> An estimated 10+ million wallets depend on that NODE_BLOOM to be
> updated. So far, I haven't heard of an alternative, except reading all
> transactions and full blocks.
>
> It goes without saying pulling the rug under that many wallets is a
> disastrous idea for the adoption of Bitcoin.
>
>> well-known DoS vectors
>
> I asked many people, even some "core developers" at meetings, but nobody
> ever was able to explain the DoS vector. I think this is just a myth.
>
> Yes, you can set an overly blurry filter and thus cause useless traffic,
> but it never exceeds just drinking from the full firehose (which this
> change doesn't prohibit). So where is the point? An attacker will just
> switch filtering off, or in fact has never used it.
>
>> It is not anticipated that
>> this will result in a significant lack of availability of
>> NODE_BLOOM-enabled nodes in the coming years
>
> Why don't you anticipate that? People almost never change defaults,
> especially if it's not for their own immediate benefit. At the same
> time, release notes in general recommend updating to the latest version.
> I *do* anticipate this will reduce the number of nodes usable by a large
> enough amount so that the feature will become unstable.
>
>> clients
>> which rely on the availability of NODE_BLOOM-supporting nodes on the
>> P2P network should consider the process of migrating
>> to a more modern (and less trustful and privacy-violating) alternative
>> over the coming years.
>
> There is no such alternative.
>
> I strongly recommend postponing this change until an alternative exists
> and then give developers enough time to implement, test and roll out.
>
> I also think as long as we don't have an alternative, we should improve
> the current filtering for segwit. E.g. testing the scripts themselves
> and each scriptPubKey spent by any input against the filter would do,
> and it also fixes the main privacy issue with server-side filtering
> (wallets have to add two items per address to the filter).
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-22 5:01 ` Matt Corallo
@ 2019-07-22 15:58 ` Justus Ranvier
2019-07-26 7:45 ` Tamas Blummer
0 siblings, 1 reply; 23+ messages in thread
From: Justus Ranvier @ 2019-07-22 15:58 UTC (permalink / raw)
To: Matt Corallo, Bitcoin Protocol Discussion, Andreas Schildbach
[-- Attachment #1.1.1: Type: text/plain, Size: 806 bytes --]
On 7/22/19 12:01 AM, Matt Corallo via bitcoin-dev wrote:
> Finally, regarding alternatives, the filter-generation code for BIP
> 157/158 has been in Bitcoin Core for some time, though the P2P serving
> side of things appears to have lost any champions working on it. I
> presume one of the Lightning folks will eventually, given they appear to
> be requiring their users connect to a handful of their own servers right
> now, but if you really need it, its likely not a ton of work to pipe
> them through.
If you want projects to adopt BIP-157/158, you'd do well to fix the
numerous errors in the specification.
As it stands right now it is impossible to implement the protocol using
the specification because he code examples are broken to the point of
appearing intentionally sabotaged.
[-- Attachment #1.1.2: 0x9B2D74E2A30A85B9.asc --]
[-- Type: application/pgp-keys, Size: 29805 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-22 15:58 ` Justus Ranvier
@ 2019-07-26 7:45 ` Tamas Blummer
0 siblings, 0 replies; 23+ messages in thread
From: Tamas Blummer @ 2019-07-26 7:45 UTC (permalink / raw)
To: Justus Ranvier, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]
Hi Justus,
It might be helpful to consult the Rust implementation of BIP158 here:
https://github.com/rust-bitcoin/rust-bitcoin/blob/master/src/util/bip158.rs
It has a cleaner structure than Core or Neutrino, includes server and client side
and passes Core's test vectors.
Regards,
Tamas Blummer
> On Jul 22, 2019, at 17:58, Justus Ranvier via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
>
> Signed PGP part
> On 7/22/19 12:01 AM, Matt Corallo via bitcoin-dev wrote:
>> Finally, regarding alternatives, the filter-generation code for BIP
>> 157/158 has been in Bitcoin Core for some time, though the P2P serving
>> side of things appears to have lost any champions working on it. I
>> presume one of the Lightning folks will eventually, given they appear to
>> be requiring their users connect to a handful of their own servers right
>> now, but if you really need it, its likely not a ton of work to pipe
>> them through.
>
> If you want projects to adopt BIP-157/158, you'd do well to fix the
> numerous errors in the specification.
>
> As it stands right now it is impossible to implement the protocol using
> the specification because he code examples are broken to the point of
> appearing intentionally sabotaged.
>
>
>
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-21 22:56 ` Andreas Schildbach
2019-07-22 5:01 ` Matt Corallo
@ 2019-07-22 8:32 ` Peter Todd
2019-07-22 13:25 ` Jonas Schnelli
` (2 subsequent siblings)
4 siblings, 0 replies; 23+ messages in thread
From: Peter Todd @ 2019-07-22 8:32 UTC (permalink / raw)
To: Andreas Schildbach, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 609 bytes --]
On Mon, Jul 22, 2019 at 12:56:33AM +0200, Andreas Schildbach via bitcoin-dev wrote:
> An estimated 10+ million wallets depend on that NODE_BLOOM to be
> updated. So far, I haven't heard of an alternative, except reading all
> transactions and full blocks.
Can you specify exactly which wallets those are?
Secondly, this doesn't stop people from running NODE_BLOOM nodes, and the DNS
seed infrastructure among others can easily direct wallets to those nodes. This
is of course not very secure, but bloom filters have never been very secure.
--
https://petertodd.org 'peter'[:-1]@petertodd.org
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-21 22:56 ` Andreas Schildbach
2019-07-22 5:01 ` Matt Corallo
2019-07-22 8:32 ` Peter Todd
@ 2019-07-22 13:25 ` Jonas Schnelli
2019-07-22 17:17 ` Luke Dashjr
2019-07-22 17:26 ` Luke Dashjr
2019-07-23 14:47 ` Andreas Schildbach
4 siblings, 1 reply; 23+ messages in thread
From: Jonas Schnelli @ 2019-07-22 13:25 UTC (permalink / raw)
To: Andreas Schildbach, Bitcoin Protocol Discussion
[-- Attachment #1.1: Type: text/plain, Size: 4286 bytes --]
Hi Andreas
>> well-known DoS vectors
>
> I asked many people, even some "core developers" at meetings, but nobody
> ever was able to explain the DoS vector. I think this is just a myth.
No. They are not a myth [1] [2] [3].
> Yes, you can set an overly blurry filter and thus cause useless traffic,
> but it never exceeds just drinking from the full firehose (which this
> change doesn't prohibit). So where is the point? An attacker will just
> switch filtering off, or in fact has never used it.
I guess it’s not about traffic DoS. It’s about asking a peer for extensive CPU and disk work. The NODE_BLOOM peers do provide this service for free while it’s not directly beneficial for the Bitcoin Network (pure consumed CPU/disk time).
>
>> It is not anticipated that
>> this will result in a significant lack of availability of
>> NODE_BLOOM-enabled nodes in the coming years
>
> Why don't you anticipate that? People almost never change defaults,
> especially if it's not for their own immediate benefit. At the same
> time, release notes in general recommend updating to the latest version.
> I *do* anticipate this will reduce the number of nodes usable by a large
> enough amount so that the feature will become unstable.
I guess this is speculation.
A quick lookup in my crawler databases shows me that there are more than 8’000 „good“ peers supporting NODE_BLOOM right now.
I don’t expect that this number drops rapidly, but maybe in the long run ("in years“, but again: speculation).
We eventually can’t expect - in the long run - that nodes provide disk/CPU intense services for free to clients not contributing back to the network.
However, sadly, due to the privacy leaks in BIP37, I expect that there will always be a wide range of peers offering NODE_BLOOM in return of collecting valuable data.
>
>> clients
>> which rely on the availability of NODE_BLOOM-supporting nodes on the
>> P2P network should consider the process of migrating
>> to a more modern (and less trustful and privacy-violating) alternative
>> over the coming years.
>
> There is no such alternative.
>
> I strongly recommend postponing this change until an alternative exists
> and then give developers enough time to implement, test and roll out.
NODE_BLOOM was added in Core 0.12 [4].
BIP111 is from 2015 [2].
One who follows the protocol development should have known that defaulting NODE_BLOOM to off was something anticipated by most developers.
I would say that there are possible alternatives, like BIP158 (though BIP157 is not widely available on the network yet).
Client side filtering works also by collecting the filter form a centralised service by the wallet provider(s) or a CDN.
You may miss transactions by a dishonest filter-packet, so may you by talking to a dishonest NODE_BLOOM peer.
Of course BIP158 is still young and – who knows – eventually once committed to consensus layer (coinbase).
> I also think as long as we don't have an alternative, we should improve
> the current filtering for segwit. E.g. testing the scripts themselves
> and each scriptPubKey spent by any input against the filter would do,
> and it also fixes the main privacy issue with server-side filtering
> (wallets have to add two items per address to the filter).
I think the consensus among protocol developers is (please speak up), that BIP37 (public server based tx filtering) – in general – was a conceptual mistake.
Maybe extending it further is the wrong step, especially when promising alternatives like BIP158 (neutrino) are around.
The fact that nobody cared about extending it for SW may also underline that BIP37 is seen as a conceptual mistake and/or "low interest in further extensions“.
/jonas
[1] https://github.com/petertodd/bloom-io-attack <https://github.com/petertodd/bloom-io-attack>
[2] https://github.com/bitcoin/bips/blob/master/bip-0111.mediawiki <https://github.com/bitcoin/bips/blob/master/bip-0111.mediawiki>
[3] https://github.com/bitcoin/bitcoin/pull/9238 <https://github.com/bitcoin/bitcoin/pull/9238>
[4] https://github.com/bitcoin/bitcoin/blob/47d981e8273804a040d71665a4cb16038d6717e1/doc/release-notes/release-notes-0.12.0.md#node_bloom-service-bit
[-- Attachment #1.2: Type: text/html, Size: 6236 bytes --]
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-22 13:25 ` Jonas Schnelli
@ 2019-07-22 17:17 ` Luke Dashjr
0 siblings, 0 replies; 23+ messages in thread
From: Luke Dashjr @ 2019-07-22 17:17 UTC (permalink / raw)
To: bitcoin-dev, Jonas Schnelli; +Cc: Andreas Schildbach
On Monday 22 July 2019 13:25:25 Jonas Schnelli via bitcoin-dev wrote:
> > I also think as long as we don't have an alternative, we should improve
> > the current filtering for segwit. E.g. testing the scripts themselves
> > and each scriptPubKey spent by any input against the filter would do,
> > and it also fixes the main privacy issue with server-side filtering
> > (wallets have to add two items per address to the filter).
>
> I think the consensus among protocol developers is (please speak up), that
> BIP37 (public server based tx filtering) – in general – was a conceptual
> mistake. Maybe extending it further is the wrong step, especially when
> promising alternatives like BIP158 (neutrino) are around.
Neutrino is very controversial, and NOT less trustful than bloom filters.
It also uses significantly more bandwidth.
It seems a better approach is to add Stratum (Electrum) support, and to limit
usage of all pseudo-SPV protocols to trusted peers.
> The fact that nobody cared about extending it for SW may also underline that
> BIP37 is seen as a conceptual mistake and/or "low interest in further
> extensions“.
Eric Lombrozo added segwit support. While it was never reviewed for Core, it
has been included and supported in Knots since v0.15.1. As I understand it,
his mSIGNA wallet also makes usage of the feature.
Luke
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-21 22:56 ` Andreas Schildbach
` (2 preceding siblings ...)
2019-07-22 13:25 ` Jonas Schnelli
@ 2019-07-22 17:26 ` Luke Dashjr
2019-07-23 14:47 ` Andreas Schildbach
4 siblings, 0 replies; 23+ messages in thread
From: Luke Dashjr @ 2019-07-22 17:26 UTC (permalink / raw)
To: bitcoin-dev, Andreas Schildbach
On Sunday 21 July 2019 22:56:33 Andreas Schildbach via bitcoin-dev wrote:
> An estimated 10+ million wallets depend on that NODE_BLOOM to be
> updated.
Where do you see this number? I think it would be useful to chart.
> So far, I haven't heard of an alternative, except reading all
> transactions and full blocks.
Electrum has a JSON-based protocol that can provide information much more
efficiently.
Currently, this does require nodes that run additional software and/or
indexes, but there are plenty of them available, and there are steps that can
be taken to improve that situation.
> > It is not anticipated that
> > this will result in a significant lack of availability of
> > NODE_BLOOM-enabled nodes in the coming years
>
> Why don't you anticipate that? People almost never change defaults,
> especially if it's not for their own immediate benefit. At the same
> time, release notes in general recommend updating to the latest version.
> I *do* anticipate this will reduce the number of nodes usable by a large
> enough amount so that the feature will become unstable.
Many users run older versions, and do not update immediately. Today, only 42%
of listening nodes are using 0.18.
(If it helps ease the concern, we can keep bloom enabled by default in Knots
longer.)
> I strongly recommend postponing this change until an alternative exists
> and then give developers enough time to implement, test and roll out.
There will be time to do so, since the functionality won't disappear
overnight.
Luke
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-21 22:56 ` Andreas Schildbach
` (3 preceding siblings ...)
2019-07-22 17:26 ` Luke Dashjr
@ 2019-07-23 14:47 ` Andreas Schildbach
2019-07-24 13:11 ` Justus Ranvier
` (4 more replies)
4 siblings, 5 replies; 23+ messages in thread
From: Andreas Schildbach @ 2019-07-23 14:47 UTC (permalink / raw)
To: bitcoin-dev
(Rather than replying individually, I try to address questions and add
my remarks in one post.)
> Finally, regarding alternatives, the filter-generation code for
> BIP 157/158 has been in Bitcoin Core for some time, though the
> P2P serving side of things appears to have lost any champions
> working on it.
BIP 157/158 is not an alternative to BIP 37:
1) It causes way too much traffic for mobile users, and likely even too
much traffic for fixed lines in not so developed parts of the world.
2) It filters blocks only. It doesn't address unconfirmed transactions.
3) Afaik, it enforces/encourages address re-use. This stems from the
fact that the server decides on the filter and in particular on the
false positive rate. On wallets with many addresses, a hardcoded filter
will be too blurry and thus each block will be matched. So wallets that
follow the "one address per incoming payment" pattern (e.g. HD wallets)
at some point will be forced to wrap their key chains back to the
beginning. If I'm wrong on this one please let me know.
4) The filters are not yet committed to the blockchain. Until that
happens we'd have to trust a server to provide correct filters.
> Can you specify exactly which wallets those are?
Bitcoin Wallet, Breadwallet, BISQ and many smaller ones.
> the DNS seed infrastructure among others can easily direct
> wallets to those nodes
Last I checked none of the seeds did. But I agree this would be nice to
have.
> We eventually can’t expect - in the long run - that nodes provide
> disk/CPU intense services for free to clients not contributing back
> to the network.
I'm disappointed to learn that supporting 10M wallets gets discredited
down to "no contribution". Each node of course supports just a number of
them, but still...
> The fact that nobody cared about extending it for SW may also
> underline that BIP37 is seen as a conceptual mistake and/or "low
> interest in further extensions“.
I suspect this is a fallacy. BIP37 doesn't need to be extended for
SegWit. See Bitcoin Wallet and Breadwallet, both support SegWit and use
the original BIP 37.
It's true however that BIP 37 could be made simpler to work with, more
future-proof and more private by simply matching output scripts.
> [1] https://github.com/petertodd/bloom-io-attack
This claims to be a proof of concept, but it's missing the concept. I
could not find a description of the attack and why is it likely to be
exploited (and why it hasn't been exploited yet).
> CPU DoS attacks are also possible
If such an attack exists, it should be easy to defend against. Filter is
using too much CPU time → disconnect and blacklist the IP for some time.
I vaguely remember the infrastructure for misbehaving peers is already
present in bitcoind.
As a side note, Coinbase just announced their 30M'th wallet. I'm
convinced this massive run into custodial wallets is caused by the
public realizing around Dec 2017 that Bitcoin fails to scale. IMHO, BIP
37 is the only scaling technology that has proven successful and could –
if supported and improved rather than choked – continue to help holding
against the bitbanks trend.
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-23 14:47 ` Andreas Schildbach
@ 2019-07-24 13:11 ` Justus Ranvier
2019-07-25 3:04 ` Luke Dashjr
` (3 subsequent siblings)
4 siblings, 0 replies; 23+ messages in thread
From: Justus Ranvier @ 2019-07-24 13:11 UTC (permalink / raw)
To: Andreas Schildbach, Bitcoin Protocol Discussion
[-- Attachment #1: Type: text/plain, Size: 633 bytes --]
On 7/23/19 9:47 AM, Andreas Schildbach via bitcoin-dev wrote:
> BIP 157/158 is not an alternative to BIP 37:
They complement each other pretty well though.
Wallets can save the deterministic GCS filters in the same way as
headers, which means blocks can be re-scanned if necessary (importing
new keys, etc) offline.
Bloom filters are good for limiting mempool bandwidth as well as
controlling the fraction of each block that is downloaded.
On the BTC chain that last point doesn't matter as so much, but on
chains with larger blocks I expect that clients will ultimately end up
using both filter schemes for exactly that reason.
[-- Attachment #2: 0x9B2D74E2A30A85B9.asc --]
[-- Type: application/pgp-keys, Size: 29350 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-23 14:47 ` Andreas Schildbach
2019-07-24 13:11 ` Justus Ranvier
@ 2019-07-25 3:04 ` Luke Dashjr
2019-07-26 10:04 ` Jonas Schnelli
` (2 subsequent siblings)
4 siblings, 0 replies; 23+ messages in thread
From: Luke Dashjr @ 2019-07-25 3:04 UTC (permalink / raw)
To: bitcoin-dev, Andreas Schildbach
On Tuesday 23 July 2019 14:47:18 Andreas Schildbach via bitcoin-dev wrote:
> > the DNS seed infrastructure among others can easily direct
> > wallets to those nodes
>
> Last I checked none of the seeds did. But I agree this would be nice to
> have.
It's supported by default in sipa's bitcoin-seeder, which many DNS seeds use.
My seed also currently requires NODE_BLOOM for *any* answers returned.
(But I don't guarantee that will remain forever.)
> As a side note, Coinbase just announced their 30M'th wallet. I'm
> convinced this massive run into custodial wallets...
Light wallets are just as bad for the network as custodial wallets.
> IMHO, BIP 37 is the only scaling technology that has proven successful and
> could – if supported and improved rather than choked – continue to help
> holding against the bitbanks trend.
Light wallets are not Bitcoin scaling. They are just trusting anonymous 3rd
parties, which is harmful to Bitcoin.
Luke
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-23 14:47 ` Andreas Schildbach
2019-07-24 13:11 ` Justus Ranvier
2019-07-25 3:04 ` Luke Dashjr
@ 2019-07-26 10:04 ` Jonas Schnelli
2019-07-27 16:10 ` Matt Corallo
2019-07-26 16:48 ` Chris
2019-07-27 19:19 ` Luke Dashjr
4 siblings, 1 reply; 23+ messages in thread
From: Jonas Schnelli @ 2019-07-26 10:04 UTC (permalink / raw)
To: Andreas Schildbach, Bitcoin Protocol Discussion
[-- Attachment #1.1: Type: text/plain, Size: 4990 bytes --]
> 1) It causes way too much traffic for mobile users, and likely even too
> much traffic for fixed lines in not so developed parts of the world.
Yes. It causes more traffic than BIP37.
Basic block filters for current last ~7 days (1008 blocks) are about 19MB (just the filters).
On top, you will probably fetch a handful of irrelevant blocks due to the FPs and due to true relevant txns.
A over-the-thumb estimation: ~25MB per week of catch-up.
If you where offline for a month: ~108MB
Thats certainly more then BIP37 BF (measured 1.6MB total traffic with android schildbach wallet restore blockchain for 8 week [7 weeks headers, 1week merkleblocks]).
But lets look at it like this: for an additional, say 25MB per week (maybe a bit more), you get the ability to filter blocks without depending on serving peers who may compromise your financial privacy.
Also, if you keep the filters, further rescans do consume the same or less bandwidth than BF BIP37.
In other words: you have the chance to potentially increase privacy by consuming bandwidth in the range of a single audio podcast per week.
I would say the job of protocol developers is protect users privacy where it’s possible (as a default).
It’s probably a debatable point wether 25MB per week of traffic is worth a potential increase in privacy, though I absolutely think 25MB/week is an acceptable tradeoff.
Saving traffic is possible by using BIP37 or stratum/electrum… but developers should make sure users are __warned about the consequences__!
Additionally, it looks like, peer operators are not endless being willing to serve – for free – a CPU/disk intense service with no benefits for the network. I would question wether a decentralised form of BIP37 is sustainable in the long run (if SPV wallet provider bootstrap a net range of NODE_BLOOM peers to make it more reliable on the network would be snake-oil).
>
> 2) It filters blocks only. It doesn't address unconfirmed transactions.
Well, unconfirmed transaction are uncertain for various reasons.
BIP158 won't allow you to filter the mempool.
But as soon as you are connected to the network, you may fetch tx with inv/getdata and pick out the relevant ones (causes also traffic).
Unclear and probably impossible with the current BIP158 specs to fetch transactions that are not in active relay and are not in a block (mempool txns, at least this is true with the current observed relay tactics).
> 3) Afaik, it enforces/encourages address re-use. This stems from the
> fact that the server decides on the filter and in particular on the
> false positive rate. On wallets with many addresses, a hardcoded filter
> will be too blurry and thus each block will be matched. So wallets that
> follow the "one address per incoming payment" pattern (e.g. HD wallets)
> at some point will be forced to wrap their key chains back to the
> beginning. If I'm wrong on this one please let me know.
I’m probably the wrong guy to ask (haven’t made the numbers) but last time I rescanned a Core wallet (in my dev branch) with block filters (and a Core wallet has >2000 addresses by default) it fetched a low and acceptable amount of false positive blocks.
(Maybe someone who made the numbers step in here.)
Though, large wallets – AFAIK – also operate badly with BIP37.
>
> 4) The filters are not yet committed to the blockchain. Until that
> happens we'd have to trust a server to provide correct filters.
I wouldn’t say so. It’s on a similar level than BIP37.
BIP37 is not – and can not – be committed to the blockchain.
You fully trust the peer that it won’t…
a) create fake unconfirmed transactions (would be the same if a BIP158 wallet would show you unconfirmed transaction)
b) lies by omission (you will miss relevant transactions, eventually swipe your wallet and loose coins)
IMO, the point b) is true for BIP37 and BIP158 (as long as not commited).
In both cases, you can reduce the trust by comparing between peers / filter-providers.
b) is conceptually solvable with a soft-fork (commitment) in BIP158 (not with BIP37).
Additionally, block-filters will, very likely, be useful for other features (load/rescan an [old] wallet on a prune peer that has the filters constructed).
There is probably no clear answer like „X is better than Y“.
Personally I would like to see developers being more honest/transparent to users about the implications of the used filtering,... and giving them choices.
Imagine a user can choose between „Electrum / BIP37 / BIP158“ depending on his needs for privacy and availability of bandwidth. Eventually also taking the future usage of this wallet (will he load old private keys, will he receive money daily, etc.) into account.
Plus, full node hardware appliances that run at home (or in a trusted environment) are solving many of these issues plus adding a bunch of great features – if done right.
/jonas
[-- Attachment #1.2: Type: text/html, Size: 15526 bytes --]
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-26 10:04 ` Jonas Schnelli
@ 2019-07-27 16:10 ` Matt Corallo
0 siblings, 0 replies; 23+ messages in thread
From: Matt Corallo @ 2019-07-27 16:10 UTC (permalink / raw)
To: Jonas Schnelli, Bitcoin Protocol Discussion; +Cc: Andreas Schildbach
[-- Attachment #1: Type: text/plain, Size: 6888 bytes --]
This conversation went off the rails somewhat. I don't think there's any immediate risk of NODE_BLOOM peers being unavailable. This is a defaults change, not a removal of the code to serve BIP 37 peers (nor would I suggest removing said code while people still want to use them - the maintenance burden isn't much). Looking at historical upgrade cycles, ignoring any other factors, there will be a large number of nodes serving NODE_BLOOM for many years.
Even more importantly, if you need them, run a node or two. As long as no one is exploiting the issues with them such a node isn't *too* expensive. Or don't, I guarantee you chainanalysis or some competitor of theirs will very very happily serve bloom-filtered clients as long as such clients want to deanonymize themselves. We already see a plurality of nodes on the network are clearly not run-of-the-mill Core nodes, many of which are likely deanonimization efforts.
In some cases BIP 137 is a replacement, in some cases, indeed, it is not. I agree at a protocol level we shouldn't be passing judgement about how users wish to interact with the Bitcoin system (aside from not putting our own, personal, effort into building such things) but that isn't what's happening here. This is an important DoS fix for the average node, and I don't really understand the argument that this is going to break existing BIP 37 wallets, but if it makes you feel any better I can run some beefy BIP 37 nodes.
Matt
> On Jul 26, 2019, at 06:04, Jonas Schnelli via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
>
>
>> 1) It causes way too much traffic for mobile users, and likely even too
>> much traffic for fixed lines in not so developed parts of the world.
>
> Yes. It causes more traffic than BIP37.
> Basic block filters for current last ~7 days (1008 blocks) are about 19MB (just the filters).
> On top, you will probably fetch a handful of irrelevant blocks due to the FPs and due to true relevant txns.
> A over-the-thumb estimation: ~25MB per week of catch-up.
> If you where offline for a month: ~108MB
>
> Thats certainly more then BIP37 BF (measured 1.6MB total traffic with android schildbach wallet restore blockchain for 8 week [7 weeks headers, 1week merkleblocks]).
>
> But lets look at it like this: for an additional, say 25MB per week (maybe a bit more), you get the ability to filter blocks without depending on serving peers who may compromise your financial privacy.
> Also, if you keep the filters, further rescans do consume the same or less bandwidth than BF BIP37.
> In other words: you have the chance to potentially increase privacy by consuming bandwidth in the range of a single audio podcast per week.
>
> I would say the job of protocol developers is protect users privacy where it’s possible (as a default).
> It’s probably a debatable point wether 25MB per week of traffic is worth a potential increase in privacy, though I absolutely think 25MB/week is an acceptable tradeoff.
> Saving traffic is possible by using BIP37 or stratum/electrum… but developers should make sure users are __warned about the consequences__!
>
> Additionally, it looks like, peer operators are not endless being willing to serve – for free – a CPU/disk intense service with no benefits for the network. I would question wether a decentralised form of BIP37 is sustainable in the long run (if SPV wallet provider bootstrap a net range of NODE_BLOOM peers to make it more reliable on the network would be snake-oil).
>
>
>>
>> 2) It filters blocks only. It doesn't address unconfirmed transactions.
>
> Well, unconfirmed transaction are uncertain for various reasons.
>
> BIP158 won't allow you to filter the mempool.
> But as soon as you are connected to the network, you may fetch tx with inv/getdata and pick out the relevant ones (causes also traffic).
> Unclear and probably impossible with the current BIP158 specs to fetch transactions that are not in active relay and are not in a block (mempool txns, at least this is true with the current observed relay tactics).
>
>
>> 3) Afaik, it enforces/encourages address re-use. This stems from the
>> fact that the server decides on the filter and in particular on the
>> false positive rate. On wallets with many addresses, a hardcoded filter
>> will be too blurry and thus each block will be matched. So wallets that
>> follow the "one address per incoming payment" pattern (e.g. HD wallets)
>> at some point will be forced to wrap their key chains back to the
>> beginning. If I'm wrong on this one please let me know.
>
> I’m probably the wrong guy to ask (haven’t made the numbers) but last time I rescanned a Core wallet (in my dev branch) with block filters (and a Core wallet has >2000 addresses by default) it fetched a low and acceptable amount of false positive blocks.
> (Maybe someone who made the numbers step in here.)
>
> Though, large wallets – AFAIK – also operate badly with BIP37.
>
>>
>> 4) The filters are not yet committed to the blockchain. Until that
>> happens we'd have to trust a server to provide correct filters.
>
> I wouldn’t say so. It’s on a similar level than BIP37.
> BIP37 is not – and can not – be committed to the blockchain.
> You fully trust the peer that it won’t…
> a) create fake unconfirmed transactions (would be the same if a BIP158 wallet would show you unconfirmed transaction)
> b) lies by omission (you will miss relevant transactions, eventually swipe your wallet and loose coins)
>
> IMO, the point b) is true for BIP37 and BIP158 (as long as not commited).
> In both cases, you can reduce the trust by comparing between peers / filter-providers.
>
> b) is conceptually solvable with a soft-fork (commitment) in BIP158 (not with BIP37).
>
> Additionally, block-filters will, very likely, be useful for other features (load/rescan an [old] wallet on a prune peer that has the filters constructed).
>
>
>
> There is probably no clear answer like „X is better than Y“.
>
> Personally I would like to see developers being more honest/transparent to users about the implications of the used filtering,... and giving them choices.
> Imagine a user can choose between „Electrum / BIP37 / BIP158“ depending on his needs for privacy and availability of bandwidth. Eventually also taking the future usage of this wallet (will he load old private keys, will he receive money daily, etc.) into account.
>
> Plus, full node hardware appliances that run at home (or in a trusted environment) are solving many of these issues plus adding a bunch of great features – if done right.
>
> /jonas
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[-- Attachment #2: Type: text/html, Size: 17831 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-23 14:47 ` Andreas Schildbach
` (2 preceding siblings ...)
2019-07-26 10:04 ` Jonas Schnelli
@ 2019-07-26 16:48 ` Chris
2019-07-27 19:19 ` Luke Dashjr
4 siblings, 0 replies; 23+ messages in thread
From: Chris @ 2019-07-26 16:48 UTC (permalink / raw)
To: bitcoin-dev
On 7/23/19 10:47 AM, Andreas Schildbach via bitcoin-dev wrote:
> 3) Afaik, it enforces/encourages address re-use. This stems from the
> fact that the server decides on the filter and in particular on the
> false positive rate. On wallets with many addresses, a hardcoded filter
> will be too blurry and thus each block will be matched. So wallets that
> follow the "one address per incoming payment" pattern (e.g. HD wallets)
> at some point will be forced to wrap their key chains back to the
> beginning. If I'm wrong on this one please let me know.
Maybe someone who knows better can confirm but I thought I read the the
fp rate on the filter was chosen assuming a wallet would average a
certain number of addresses (ie, they assumed use of an HD wallet). Your
criticism might be valid for wallets that well exceed the average.
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default
2019-07-23 14:47 ` Andreas Schildbach
` (3 preceding siblings ...)
2019-07-26 16:48 ` Chris
@ 2019-07-27 19:19 ` Luke Dashjr
4 siblings, 0 replies; 23+ messages in thread
From: Luke Dashjr @ 2019-07-27 19:19 UTC (permalink / raw)
To: bitcoin-dev, Andreas Schildbach
On Tuesday 23 July 2019 14:47:18 Andreas Schildbach via bitcoin-dev wrote:
> 3) Afaik, it enforces/encourages address re-use. This stems from the
> fact that the server decides on the filter and in particular on the
> false positive rate. On wallets with many addresses, a hardcoded filter
> will be too blurry and thus each block will be matched. So wallets that
> follow the "one address per incoming payment" pattern (e.g. HD wallets)
> at some point will be forced to wrap their key chains back to the
> beginning. If I'm wrong on this one please let me know.
BTW, you are indeed wrong on this. You don't need to match every single
address the wallet has ever used, only outstanding addresses that haven't
been paid. ;)
Luke
^ permalink raw reply [flat|nested] 23+ messages in thread