Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)

From: Eric Lombrozo <elombrozo@gmail•com>
To: "Tamas Blummer" <tamas@bitsofproof•com>,
	"Jorge Timón" <jtimon@jtimon•cc>,
	"Matt Corallo" <lf-lists@mattcorallo•com>
Cc: Bitcoin Dev <bitcoin-dev@lists•linuxfoundation.org>,
	Libbitcoin <libbitcoin@lists•dyne.org>
Subject: Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks)
Date: Sun, 23 Aug 2015 02:19:26 +0000	[thread overview]
Message-ID: <CABr1YTce7Q_=J9DVsmGQYUd6OVODiEqfFi+jPM6pMYn9uNpJOA@mail.gmail.com> (raw)
In-Reply-To: <CABr1YTcx4V0Q4-ZQBEiNG-z1NKeFzxzhMekRN3YRRLz+bme2iw@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 7281 bytes --]

One thing it occurs to me (and I don't know if this has been suggested
before) we could do is separate the BIP process into at several distinct
areas:

1) Commit structure changes/consensus rule change proposals
- Consensus-building process (how are proposals debated, improved, vetted,
and selected)
- Update/deployment mechanisms for rule changes
- Specific hard fork proposals
- Specific soft fork proposals

2) Peer policies
- Seeding and discovery mechanisms
- Relay policies
- p2p message support

3) RPC

4) Everything else

On Sat, Aug 22, 2015, 6:28 PM Eric Lombrozo <elombrozo@gmail•com> wrote:

> I've been pushing for greater modularization since I first got into
> bitcoin. I got quickly frustrated when I was only able to get through very
> few things (i.e. moving core structure serialization classes to a separate
> unit not called main). Working on Bitcoin has an added layer of frustration
> that goes beyond most open source projects: even though we're clearly in
> userland working at the application layer, a good layered protocol design
> is still lacking. We have no standards process separate from what basically
> amount to updates to one specific reference implementation. And we all need
> to agree on any major change, since a blockchain that is easily forked in
> contentious ways pretty much defeats its own purpose.
>
> I went off to develop my own stack, where I could more easily avoid
> politics and focus on engineering. But I now understand the politics are
> inevitable. Bitcoin is inherently a cooperative project. Several people
> have poured themselves passionately into the reference codebase, most of
> whom did it (at least initially) purely as unpaid volunteers. There's a lot
> of love that's gone into this. But it's become pretty clear that the
> modularization is no longer merely a matter of good engineering - it is
> essential to resolving serious political challenges.
>
> Perhaps the most frustrating thing of all is watching people pushing for
> relatively superficial yet highly controversial changes while we still lack
> the proper infrastructure to handle these kinds of divergences of opinion
> without either stagnating or becoming polarized.
>
> I could continue working to reimplement an entire stack from scratch, as
> several others have also done - but besides the serious effort duplication
> this entails, it doesn't really seem like it will ultimately be a
> convergent process. It's too easy to let ego and habit dictate one's
> preferences rather than rational engineering considerations.
>
> I know that some might feel I'm just preaching to the choir, but we should
> probably take a step back from implementation hackery and try to specify
> some core protocol layers, focusing on interfaces. Specifically, we need a
> consensus layer that doesn't try to specify networking, storage, wallets,
> UI, etc. Let different people improve upon these things independently in
> their own implementations. What matters is that we all converge on a common
> history and state. At the same time, let's open up more competition on all
> these other things that are separate from the consensus layer.
>
> If only we were to dedicate a fraction of the effort we've put into this
> whole block size circus into what's actually important...and I blame myself
> as well...
>
> On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation.org> wrote:
>
>> On Aug 21, 2015, at 21:46, Jorge Timón <jtimon@jtimon•cc> wrote:
>>
>> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof•com>
>> wrote:
>>
>> Every re-implementation, re-factoring even copy-paste introduces a risk
>> of disagreement,
>> but also open the chance of doing the work better, in the sense of
>> software engineering.
>>
>>
>> But you don't want something better, you want something functionally
>> identical.
>> You may want to watch sipa's explanation on why "the implementation is
>> the specification" and the reasons to separate libconsensus:
>> https://youtu.be/l3O4nh79CUU?t=764
>>
>>
>> I do want something better, but not for the focus you have.
>>
>> Not because what you produce was not high quality, but because quality is
>> achieved at a very
>> high cost and is hard to uphold over generations of developer. You focus
>> on a single use case
>> while there are many out there for distributed ledgers.
>>
>> I think in an infrastructure for enterprise applications, building
>> consensus on the ledger is a
>> cornerstone there, but is only a piece of the solution. I built several
>> commercially successful
>> deployments where I delegated the consensus building to a border router,
>> a Bitcoin Core,
>> then interfaced that trusted peer with my  implementation that accepted
>> Core’s decisions
>> in an SPV manner. One might think of this setup as wasteful and
>> unsuitable for “small devices”
>> therefore an example of centralization people here try to avoid.
>>
>> Enterprises have sufficient resources. Solving the business problem is
>> valuable to them even at
>> magnitudes higher cost than a hobbyist would bear.
>>
>> For mainstream adoption you need to get enterprises on board too, and
>>  that is what I care of.
>> Enterprises want code that is not only high quality, but is easy to
>> maintain with a development
>> team with high attrition. One has to take whatever help is offered for
>> that, and one is modern
>> languages and runtimes.
>>
>> Bits of Proof’s own implementation of the scripts was not practically
>> relevant in my commercially
>> successful deployments, because of the use of a border router, but it
>> helped development,
>> enabling easier debug and precise error feedback esp. end even after Core
>> had a reject message.
>>
>> I integrated libconsensus only for the hope that is significantly fastens
>> application side tx verification,
>>  which it has turned out it does not, until secp265k1 is integrated.
>>
>> I would likely use an other extended libconsensus too, but do not think
>> there was a dependency on
>> that for enterprise development.
>>
>> It would help there more to have a slim protocol server, no wallet, no
>> rpc, no qt but a high
>> performance remoting API.
>>
>> Since you already depend on libconsensus for VerifyScript, wouldn't it
>> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
>> You would still have complete control over storage, concurrency,
>> networking, policy...
>> My plan is for the C API to interface with the external storage by
>> passing a function pointer to it.
>>
>>
>> Storage and validation is non-trivially interconnected, but I now the
>> separation can be done,
>> since I did it.
>>
>> Excuse me, but function pointers is a pattern I used in the 80’s. I know
>> that they are behind
>> the curtain of modern abstractions with similar use, I still prefer not
>> to see them again.
>>
>> Tamas Blummer
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>

[-- Attachment #2: Type: text/html, Size: 15088 bytes --]