Re: [bitcoin-dev] Using a storage engine without UTXO-index

From: Tomas <tomas@tomasvdw•nl>
To: Greg Sanders <gsanders87@gmail•com>,
	Bitcoin Protocol Discussion
	<bitcoin-dev@lists•linuxfoundation.org>
Subject: Re: [bitcoin-dev] Using a storage engine without UTXO-index
Date: Fri, 07 Apr 2017 18:02:35 +0200	[thread overview]
Message-ID: <1491580955.73830.937628696.4EEA507C@webmail.messagingengine.com> (raw)
In-Reply-To: <CAB3F3DvG4NSBEk1vS-KWQjn3PXWOwF3xNP7_txUpwTdDMHettg@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 4114 bytes --]

Thank you,

The benches are running in Google Cloud Engine; currently on 8 vCPU
32gb, but I tend to switch hardware regularly.

Roughly, the results are better for Bitcrust with high end hardware and
the difference for total block validations is mostly diminished at 2
vCPU, 7,5 gb.

Note that the spend-tree optimization primarily aims to improve peak
load order validation; when a block with pre-synced transactions comes
in, but this is tricky to accurately bench with Core using this simple
method of comparison by logs.

I will upgrade to, and show the results against 0.14 in the next weeks.

Best,

Tomas

On Fri, Apr 7, 2017, at 16:14, Greg Sanders wrote:

> Interesting work.

> 

> I was wondering if you could tellank  us what specs for the machine
> being used as preliminary benchmark is here:
> https://bitcrust.org/results ?
> 

> I'd be interested to also see comparisons with 0.14 which has some
> improvements for script validation with more cores.
> 

> On Fri, Apr 7, 2017 at 4:47 AM, Tomas via bitcoin-dev <bitcoin-
> dev@lists•linuxfoundation.org> wrote:
>> Thank you Marcos,

>> 

>>  Though written in Rust, bitcrust-db is definitely usable as
>>  pluggable
>>  module as its interface will be roughly some queries, add_tx and

>>  add_block with blobs and flags. (Bitcrust internally uses a

>>  deserialize-only model, keeping references to the blobs with the
>>  parsed
>>  data).

>> 

>>  However, from Core's side I believe network and storage are
>>  currently
>>  rather tightly coupled, which will make this far from trivial.

>> 

>>  Regardless, I am also hoping (with funding & a team) to build a
>>  Bitcrust
>>  networking component as well to bring a strong competitor to the
>>  market.
>> 

>>  best,

>>  Tomas

>> 

>> 

>> 

>> 

>> On Fri, Apr 7, 2017, at 09:55, Marcos mayorga wrote:

>>  > Hi Tomas,

>>  >

>>  > I've read it and think it is an excellent work, I'd like to see it
>>  > integrated into bitcoin-core as a 'kernel module'.

>>  >

>>  > I see there are a lot of proof of concepts out there, IMO
>>  > every one
>>  > deserve a room in the bitcoin client as a selectable feature, to
>>  > make the
>>  > software more flexible and less dictatorial, an user could easily
>>  > select
>>  > which features she wants to run.

>>  >

>>  > Best regards,

>>  > Marcos

>>  >

>>  > > I have been working on a bitcoin implementation that uses a
>>  > > different
>>  > > approach to indexing for verifying the order of transactions.
>>  > > Instead of
>>  > > using an index of unspent outputs, double spends are verified by
>>  > > using a
>>  > > spend-tree where spends are scanned against spent outputs
>>  > > instead of
>>  > > unspent outputs.

>>  > >

>>  > > This allows for much better concurrency, as not only blocks, but
>>  > > also
>>  > > individual inputs can be verified fully in parallel.

>>  > >

>>  > > I explain the approach at https://bitcrust.org, source code is
>>  > > available
>>  > > at https://github.com/tomasvdw/bitcrust

>>  > >

>>  > > I am sharing this not only to ask for your feedback, but also to
>>  > > call
>>  > > for a clear separation of protocol and implementations: As this
>>  > > solution, reversing the costs of outputs and inputs, seems to
>>  > > have
>>  > > excellent performance characteristics (as shown in the test
>>  > > results),
>>  > > updates to the protocol addressing the UTXO growth, might not be
>>  > > worth
>>  > > considering *protocol improvements* and it might be best to
>>  > > address
>>  > > these concerns as implementation details.

>>  > >

>>  > > Kind regards,

>>  > > Tomas van der Wansem

>>  > > tomas@bitcrust•org

>>  > > Bitcrust

>>  > > _______________________________________________

>>  > > bitcoin-dev mailing list

>>  > > bitcoin-dev@lists•linuxfoundation.org

>>  > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

>>  > >

>>  >

>>  >

>>  _______________________________________________

>>  bitcoin-dev mailing list

>> bitcoin-dev@lists•linuxfoundation.org

>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 6443 bytes --]