From: Troy Benjegerdes <hozer@hozed•org>
To: Jeff Garzik <jgarzik@bitpay•com>
Cc: Bitcoin Dev <bitcoin-development@lists•sourceforge.net>
Subject: Re: [Bitcoin-development] Open development processes and reddit charms
Date: Tue, 16 Dec 2014 13:24:15 -0600 [thread overview]
Message-ID: <20141216192415.GQ29130@nl.grid.coop> (raw)
In-Reply-To: <CAJHLa0M0YeEWWaP4gNpwnaDq56w=AHW_kmLr=f3AVvHDB89SNA@mail.gmail.com>
Thank you Jeff.
Having looked at a lot of linux code, and now a lot of bitcoin code, the
biggest long-term systemic risk I see is that Bitcoin has is the lack of
code janitors.
The problem is that janitoring was *disruptive* for non-x86 linux architectures
when it first got going, and it's going to be very disruptive for bitcoin as
well, but it **needs** to happen.
The code is too complex and hard to follow as it is now. (now, I could just
be speaking because I haven't paid the social debt of looking at the latest
bitcoin code, including libconsensus), but there really needs to be a focus
on readability, maintainability, and (as much as I hate to say it) a rather
hard-line policy on coding standards.
I don't care which tabbing style or column width you pick, but **pick one**,
and enforce it across the entire codebase.
Maybe this should be bitcoin-stable, and bitcoin-devel, with a 6-9 month
social expectation of minimal cosmetic changes in -stable, with a 1 month
'merge window' where -devel turns into -stable.
On Tue, Dec 16, 2014 at 12:59:06PM -0500, Jeff Garzik wrote:
> It can be useful to review open source development processes from time to
> time. This reddit thread[1] serves use both as a case study, and also a
> moment of OSS process introduction for newbies.
> [1]
> http://www.reddit.com/r/Bitcoin/comments/2pd0zy/peter_todd_is_saying_shoddy_development_on_v010/
>
>
>
>
> *Dirty Laundry*
> When building businesses or commercial software projects, outsiders
> typically hear little about the internals of project development. The
> public only hears what the companies release, which is prepped and
> polished. Internal disagreements, schedule slips, engineer fistfights are
> all unseen.
>
> Open source development is the opposite. The goal is radical
> transparency. Inevitably there is private chatter (0day bugs etc.), but
> the default is openness. This means that is it normal practice to "air
> dirty laundry in public." Engineers will disagree, sometimes quietly,
> sometimes loudly, sometimes rudely and with ad hominem attacks. On the
> Internet, there is a pile-on effect, where informed and uninformed
> supporters add their 0.02 BTC.
>
> Competing interests cloud the issues further. Engineers are typically
> employed by an organization, as a technology matures. Those organizations
> have different strategies and motivations. These organizations will
> sponsor work they find beneficial. Sometimes those orgs are non-profit
> foundations, sometimes for-profit corporations. Sometimes that work is
> maintenance ("keep it running"), sometimes that work is developing new,
> competitive features that company feels will give it a better market
> position. In a transparent development environment, all parties are
> hyperaware of these competing interests. Internet natterers painstakingly
> document and repeat every conspiracy theory about Bitcoin Foundation,
> Blockstream, BitPay, various altcoin developers, and more as a result of
> these competing interests.
>
> Bitcoin and altcoin development adds an interesting new dimension.
> Sometimes engineers have a more direct conflict of interest, in that the
> technology they are developing is also potentially their road to instant
> $millions. Investors, amateur and professional, have direct stakes in a
> certain coin or coin technology. Engineers also have an emotional stake in
> technology they design and nurture. This results in incentives where
> supporters of a non-bitcoin technology work very hard to thump bitcoin.
> And vice versa. Even inside bitcoin, you see "tree chains vs. side chains"
> threads of a similar stripe. This can lead to a very skewed debate.
>
> That should not distract from the engineering discussion. Starting from
> first principles, Assume Good Faith[2]. Most engineers in open source tend
> to mean what they say. Typically they speak for themselves first, and
> their employers value that engineer's freedom of opinion. Pay attention to
> the engineers actually working on the technology, and less attention to the
> noise bubbling around the Internet like the kindergarten game of grapevine.
> [2] http://en.wikipedia.org/wiki/Wikipedia:Assume_good_faith
>
> Being open and transparent means engineering disagreements happen in
> public. This is normal. Open source engineers live an aquarium life[3].
> [3] https://www.youtube.com/watch?v=QKe-aO44R7k
>
>
>
>
> *What the fork?*
> In this case, a tweet suggests consensus bug risks, which reddit account
> "treeorsidechains" hyperbolizes into a dramatic headline[1]. However, the
> headline would seem to be the opposite of the truth. Several changes were
> merged during 0.10 development which move snippets of source code into new
> files and new sub-directories. The general direction of this work is
> creating a "libconsensus" library that carefully encapsulates consensus
> code in a manner usable by external projects. This is a good thing.
>
> The development was performed quite responsible: Multiple developers would
> verify each cosmetic change, ensuring no behavior changes had been
> accidentally (or maliciously!) introduced. Each pull request receives a
> full multi-platform build + automated testing, over and above individual
> dev testing. Comparisons at the assembly language level were sometimes
> made in critical areas, to ensure zero before-and-after change. Each
> transformation gets the Bitcoin Core codebase to a more sustainable, more
> reusable state.
>
> Certainly zero-change is the most conservative approach. Strictly speaking,
> that has the lowest consensus risk. But that is a short term mentality.
> Both Bitcoin Core and the larger ecosystem will benefit when the "hairball"
> pile of source code is cleaned up. Progress has been made on that front in
> the past 2 years, and continues. *Long term*, combined with the
> "libconsensus" work, that leads to less community-wide risk.
>
> The key is balance. Continue software engineering practices -- like those
> just mentioned above -- that enable change with least consensus risk. Part
> of those practices is review at each step of the development process:
> social media thought bubble, mailing list post, pull request, git merge,
> pre-release & release. It probably seems chaotic at times. In effect,
> git[hub] and the Internet enable a dynamic system of review and feedback,
> where each stage provides a check-and-balance for bad ideas and bad
> software changes. It's a human process, designed to acknowledge and handle
> that human engineers are fallible and might make mistakes (or be
> coerced/under duress!). History and field experience will be the ultimate
> judge, but I think Bitcoin Core is doing good on this score, all things
> considered.
>
> At the end of the day, while no change is without risk, version 0.10 work
> was done with attention to consensus risk at multiple levels (not just
> short term).
>
>
>
>
> *Technical and social debt*
> Working on the Linux kernel was an interesting experience that combined
> git-driven parallel development and a similar source code hairball. One of
> the things that quickly became apparent is that cosmetic patches,
> especially code movement, was hugely disruptive. Some even termed it
> anti-social. To understand why, it is important to consider how modern
> software changes are developed:
>
> Developers work in parallel on their personal computers to develop XYZ
> change, then submit their change "upstream" as a github pull request. Then
> time passes. If code movement and refactoring changes are accepted
> upstream before XYZ, then the developer is forced update XYZ -- typically
> trivial fixes, re-review XYZ, and re-test XYZ to ensure it remains in a
> known-working state.
>
> Seemingly cosmetic changes such as code movement have a ripple effect on
> participating developers, and wider developer community. Every developer
> who is *not* immediately merged upstream must bear the costs of updating
> their unmerged work.
>
> Normally, this is expected. Encouraging developers to build on top of
> "upstream" produces virtuous cycles.
>
> However, a constant stream of code movement and cosmetic changes may
> produce a constant stream of disruption to developers working on
> non-trivial features that take a bit longer to develop before going
> upstream. Trivial changes are encouraged, and non-trivial changes face a
> binary choice of (a) be merged immediately or (b) bear added re-base,
> re-view, re-test costs.
>
> Taken over a timescale of months, I argue that a steady stream of cosmetic
> code movement changes serves as a disincentive to developers working with
> upstream. Each upstream breakage has a ripple effect to all developers
> downstream, and imposes some added chance of newly introduced bugs on
> downstream developers. I'll call this "social debt", a sort of technical
> debt[4] for developers.
> [4] http://en.wikipedia.org/wiki/Technical_debt
>
> As mentioned above, the libconsensus and code movement work is a net gain.
> The codebase needs cleaning up. Each change however incurs a little bit of
> social debt. Life is a little bit harder on people trying to get work into
> the tree. Developers are a little bit more discouraged at the busy-work
> they must perform. Non-trivial pull requests take a little bit longer to
> approve, because they take a little bit more work to rebase (again).
>
> A steady flow of code movement and cosmetic breakage into the tree may be a
> net gain, but it also incurs a *lot* of social debt. In such situations,
> developers find that tested, working out-of-tree code repeatedly stops
> working *during the process of trying to get that work in-tree*. Taken
> over time, it discourages working on the tree. It is rational to sit back,
> *not* work on the tree, let the breakage stop, and then pick up the pieces.
>
>
>
>
> *Paradox Unwound*
> Bitcoin Core, then, is pulled in opposite directions by a familiar
> problem. It is generally agreed that the codebase needs further
> refactoring. That's not just isolated engineer nit-picking. However, for
> non-trivial projects, refactoring is always anti-social in the short term.
> It impacts projects other than your own, projects you don't even know
> about. One change causes work for N developers. Given these twin opposing
> goals, the key, as ever, is finding the right balance.
>
> Much like "feature freeze" in other software projects, developing a policy
> that opens and closes windows for code movement and major disruptive
> changes seems prudent. One week of code movement & cosmetics followed by 3
> weeks without, for example. Part of open source parallel development
> is *social
> signalling*: Signal to developers when certain changes are favored or not,
> then trust they can handle the rest from there.
>
> While recent code movement commits themselves are individually ACK-worthy,
> professionally executed and moving towards a positive goal, I think the
> project could strike a better balance when it comes to disruptive cosmetic
> changes, a balance that better encourages developers to work on more
> involved Bitcoin Core projects.
next prev parent reply other threads:[~2014-12-16 20:04 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-16 17:59 Jeff Garzik
2014-12-16 19:24 ` Troy Benjegerdes [this message]
2014-12-17 8:53 ` Wladimir
2014-12-18 5:56 ` 21E14
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141216192415.GQ29130@nl.grid.coop \
--to=hozer@hozed$(echo .)org \
--cc=bitcoin-development@lists$(echo .)sourceforge.net \
--cc=jgarzik@bitpay$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox