Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"

From: Jeff Garzik <jgarzik@gmail•com>
To: Tier Nolan <tier.nolan@gmail•com>
Cc: Bitcoin Dev <bitcoin-dev@lists•linuxfoundation.org>
Subject: Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression"
Date: Tue, 10 Nov 2015 11:46:15 -0500	[thread overview]
Message-ID: <CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com> (raw)
In-Reply-To: <CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 3626 bytes --]

Comments:

1) cblock seems a reasonable way to extend the protocol.  Further wrapping
should probably be done at the stream level.

2) zlib has crappy security track record.

3) A fallback path to non-compressed is required, should compression fail
or crash.

4) Most blocks and transactions have runs of zeroes and/or highly common
bit-patterns, which contributes to useful compression even at smaller
sizes.  Peter Ts's most recent numbers bear this out.  zlib has a
dictionary (32K?) which works well with repeated patterns such as those you
see with concatenated runs of transactions.

5) LZO should provide much better compression, at a cost of CPU performance
and using a less-reviewed, less-field-tested library.

On Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

>
>
> On Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper <
> peter.tschipper@gmail•com> wrote:
>
>> There are better ways of sending new blocks, that's certainly true but
>> for sending historical blocks and seding transactions I don't think so.
>> This PR is really designed to save bandwidth and not intended to be a huge
>> performance improvement in terms of time spent sending.
>>
>
> If the main point is for historical data, then sticking to just blocks is
> the best plan.
>
> Since small blocks don't compress well, you could define a "cblocks"
> message that handles multiple blocks (just concatenate the block messages
> as payload before compression).
>
> The sending peer could combine blocks so that each cblock is compressing
> at least 10kB of block data (or whatever is optimal).  It is probably worth
> specifying a maximum size for network buffer reasons (either 1MB or 1 block
> maximum).
>
> Similarly, transactions could be combined together and compressed "ctxs".
> The inv messages could be modified so that you can request groups of 10-20
> transactions.  That would depend on how much of an improvement compressed
> transactions would represent.
>
> More generally, you could define a message which is a compressed message
> holder.  That is probably to complex to be worth the effort though.
>
>
>
>>
>> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev <
>> <bitcoin-dev@lists•linuxfoundation.org>
>> bitcoin-dev@lists•linuxfoundation.org> wrote:
>>
>>> On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev <
>>> bitcoin-dev@lists•linuxfoundation.org> wrote:
>>>
>>>
>>>> I think 25% bandwidth savings is certainly considerable, especially for
>>>> people running full nodes in countries like Australia where internet
>>>> bandwidth is lower and there are data caps.
>>>>
>>>
>>> This reinforces the idea that such trade-off decisions should be be
>>> local and negotiated between peers, not a required feature of the network
>>> P2P.
>>>
>>>
>>> --
>>> Johnathan Corgan
>>> Corgan Labs - SDR Training and Development Services
>>> http://corganlabs.com
>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists•linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing listbitcoin-dev@lists•linuxfoundation.orghttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 7807 bytes --]