Re: [bitcoin-dev] Using a storage engine without UTXO-index

From: Troy Benjegerdes <hozer@hozed•org>
To: Bram Cohen <bram@bittorrent•com>,
	Bitcoin Protocol Discussion
	<bitcoin-dev@lists•linuxfoundation.org>
Subject: Re: [bitcoin-dev] Using a storage engine without UTXO-index
Date: Sat, 8 Apr 2017 21:22:11 +0000	[thread overview]
Message-ID: <20170408212211.GG23538@hostname.unassigned> (raw)
In-Reply-To: <CA+KqGko0cDY29bhznMxJJ7yAUTuB6GaDDNGBRwzssJUxM_53xQ@mail.gmail.com>

I would advise anyone worried about 'hard drive access' to order a
512GB NVME (pci-express interface) flash drive (or a laptop), and
I expect the performance will make you wonder why you ever bothered
with cloud.

My (very brief) analysis of the performance of a full chain download
on a new laptop was that there was more overhead in lock contention and
database writes and it barely loaded the machine. Now maybe this is
because the flash **write** speed is slow (but still several orders
of magnitude above spinning disk), but random reads are sure blazing
fast.

Flash storage sizes also appear to be growing at similiar rates as the
total blockchain size.

Which begs another question: In a distributed byzantine fault-tolerant
system, why do we even need to bother with persistant storage, except
for long-term archival and chain of custody issues, which we could 
serialize the in-memory structures out as a stream to things like tape
drives or write-once optical media.

On Fri, Apr 07, 2017 at 11:39:18AM -0700, Bram Cohen via bitcoin-dev wrote:
> Expanding on this question a bit, it's optimized for parallel access, but
> hard drive access isn't parallel and memory accesses are very fast, so
> shouldn't the target of optimization be about cramming as much as possible
> in memory and minimizing disk accesses?
> 
> On Fri, Apr 7, 2017 at 11:18 AM, Gregory Maxwell via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation.org> wrote:
> 
> > On Thu, Apr 6, 2017 at 10:12 PM, Tomas via bitcoin-dev
> > <bitcoin-dev@lists•linuxfoundation.org> wrote:
> > >As this
> > > solution, reversing the costs of outputs and inputs, seems to have
> > > excellent performance characteristics (as shown in the test results),
> > > updates to the protocol addressing the UTXO growth, might not be worth
> > > considering *protocol improvements*
> >
> > I'm still lost on this-- AFAICT your proposals long term resource
> > requirements are directly proportional to the amount of unspent output
> > data, which grows over time at some fraction of the total transaction
> > volume (plus the rate of spending which is more or less a constant).
> >
> > Can you help out my understanding here?
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists•linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> >

> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev