Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"

From: Peter Tschipper <peter.tschipper@gmail•com>
To: bitcoin-dev@lists•linuxfoundation.org
Subject: Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"
Date: Tue, 10 Nov 2015 09:09:06 -0800	[thread overview]
Message-ID: <564224B2.9090903@gmail.com> (raw)
In-Reply-To: <CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 4893 bytes --]

On 10/11/2015 8:46 AM, Jeff Garzik via bitcoin-dev wrote:
> Comments:
>
> 1) cblock seems a reasonable way to extend the protocol.  Further
> wrapping should probably be done at the stream level.
agreed.
>
> 2) zlib has crappy security track record.
>
Zlib had a bad buffer overflow bug but that was in 2005 and it got a lot
of press at the time.  It's was fixed in version 1.2.3...we're on 1.2.8
now.  I'm not aware of any other current issues with zlib. Do you have a
citation?

> 3) A fallback path to non-compressed is required, should compression
> fail or crash.
agreed.
>
> 4) Most blocks and transactions have runs of zeroes and/or highly
> common bit-patterns, which contributes to useful compression even at
> smaller sizes.  Peter Ts's most recent numbers bear this out.  zlib
> has a dictionary (32K?) which works well with repeated patterns such
> as those you see with concatenated runs of transactions.
>
> 5) LZO should provide much better compression, at a cost of CPU
> performance and using a less-reviewed, less-field-tested library.
I don't think LZO will give as good compression here but I will do some
benchmarking when I can.

>
>
>
>
>
> On Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via bitcoin-dev
> <bitcoin-dev@lists•linuxfoundation.org
> <mailto:bitcoin-dev@lists•linuxfoundation.org>> wrote:
>
>
>
>     On Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper
>     <peter.tschipper@gmail•com <mailto:peter.tschipper@gmail•com>> wrote:
>
>         There are better ways of sending new blocks, that's certainly
>         true but for sending historical blocks and seding transactions
>         I don't think so.  This PR is really designed to save
>         bandwidth and not intended to be a huge performance
>         improvement in terms of time spent sending.
>
>
>     If the main point is for historical data, then sticking to just
>     blocks is the best plan.
>
>     Since small blocks don't compress well, you could define a
>     "cblocks" message that handles multiple blocks (just concatenate
>     the block messages as payload before compression). 
>
>     The sending peer could combine blocks so that each cblock is
>     compressing at least 10kB of block data (or whatever is optimal). 
>     It is probably worth specifying a maximum size for network buffer
>     reasons (either 1MB or 1 block maximum).
>
>     Similarly, transactions could be combined together and compressed
>     "ctxs".  The inv messages could be modified so that you can
>     request groups of 10-20 transactions.  That would depend on how
>     much of an improvement compressed transactions would represent.
>
>     More generally, you could define a message which is a compressed
>     message holder.  That is probably to complex to be worth the
>     effort though.
>
>      
>
>>
>>         On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via
>>         bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org
>>         <mailto:bitcoin-dev@lists•linuxfoundation.org>> wrote:
>>
>>             On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev
>>             <bitcoin-dev@lists•linuxfoundation.org
>>             <mailto:bitcoin-dev@lists•linuxfoundation.org>> wrote:
>>              
>>
>>                 I think 25% bandwidth savings is certainly
>>                 considerable, especially for people running full
>>                 nodes in countries like Australia where internet
>>                 bandwidth is lower and there are data caps.
>>
>>
>>              This reinforces the idea that such trade-off decisions
>>             should be be local and negotiated between peers, not a
>>             required feature of the network P2P.
>>              
>>
>>             -- 
>>             Johnathan Corgan
>>             Corgan Labs - SDR Training and Development Services
>>             http://corganlabs.com
>>
>>             _______________________________________________
>>             bitcoin-dev mailing list
>>             bitcoin-dev@lists•linuxfoundation.org
>>             <mailto:bitcoin-dev@lists•linuxfoundation.org>
>>             https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>>
>>         _______________________________________________
>>         bitcoin-dev mailing list
>>         bitcoin-dev@lists•linuxfoundation.org
>>         <mailto:bitcoin-dev@lists•linuxfoundation.org>
>>         https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>
>     _______________________________________________
>     bitcoin-dev mailing list
>     bitcoin-dev@lists•linuxfoundation.org
>     <mailto:bitcoin-dev@lists•linuxfoundation.org>
>     https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


[-- Attachment #2: Type: text/html, Size: 15022 bytes --]