* [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db
@ 2015-10-22 21:26 Jeff Garzik
2015-10-22 21:54 ` Patrick Strateman
` (4 more replies)
0 siblings, 5 replies; 36+ messages in thread
From: Jeff Garzik @ 2015-10-22 21:26 UTC (permalink / raw)
To: Bitcoin development mailing list
[-- Attachment #1: Type: text/plain, Size: 306 bytes --]
Here is the beginnings of an implementation to replace leveldb with sqlite:
https://github.com/jgarzik/bitcoin/tree/2015_sqlite
It builds, but still needs work before passing tests.
It was noted that leveldb is unmaintained, and this is part of researching
alternatives that are maintained and reliable.
[-- Attachment #2: Type: text/html, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 36+ messages in thread* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-22 21:26 [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db Jeff Garzik @ 2015-10-22 21:54 ` Patrick Strateman 2015-10-22 21:56 ` Joseph Gleason ⑈ ` (3 subsequent siblings) 4 siblings, 0 replies; 36+ messages in thread From: Patrick Strateman @ 2015-10-22 21:54 UTC (permalink / raw) To: bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 615 bytes --] Benchmarks? I cant imagine that's very fast. On 10/22/2015 02:26 PM, Jeff Garzik via bitcoin-dev wrote: > Here is the beginnings of an implementation to replace leveldb with > sqlite: https://github.com/jgarzik/bitcoin/tree/2015_sqlite > > It builds, but still needs work before passing tests. > > It was noted that leveldb is unmaintained, and this is part of > researching alternatives that are maintained and reliable. > > > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [-- Attachment #2: Type: text/html, Size: 1606 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-22 21:26 [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db Jeff Garzik 2015-10-22 21:54 ` Patrick Strateman @ 2015-10-22 21:56 ` Joseph Gleason ⑈ 2015-10-23 6:53 ` Jonas Schnelli ` (2 subsequent siblings) 4 siblings, 0 replies; 36+ messages in thread From: Joseph Gleason ⑈ @ 2015-10-22 21:56 UTC (permalink / raw) To: Jeff Garzik, Bitcoin development mailing list [-- Attachment #1: Type: text/plain, Size: 933 bytes --] I have done a lot of recent work on local key value stores, mostly for a java electrum server I am working on. I'd suggest considering LMDB. One downside is that it is memory mapped so 32-bit systems that need over 2gb of storage are right out. Other than that, it is quite fast and seems reliable in my testing. On Thu, Oct 22, 2015 at 2:37 PM Jeff Garzik via bitcoin-dev < bitcoin-dev@lists•linuxfoundation.org> wrote: > Here is the beginnings of an implementation to replace leveldb with > sqlite: https://github.com/jgarzik/bitcoin/tree/2015_sqlite > > It builds, but still needs work before passing tests. > > It was noted that leveldb is unmaintained, and this is part of researching > alternatives that are maintained and reliable. > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > [-- Attachment #2: Type: text/html, Size: 1597 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-22 21:26 [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db Jeff Garzik 2015-10-22 21:54 ` Patrick Strateman 2015-10-22 21:56 ` Joseph Gleason ⑈ @ 2015-10-23 6:53 ` Jonas Schnelli 2015-10-23 7:45 ` Lucas Betschart 2015-10-23 10:30 ` Tom Zander 4 siblings, 0 replies; 36+ messages in thread From: Jonas Schnelli @ 2015-10-23 6:53 UTC (permalink / raw) To: Jeff Garzik, Bitcoin development mailing list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > Here is the beginnings of an implementation to replace leveldb > with sqlite: https://github.com/jgarzik/bitcoin/tree/2015_sqlite > > It builds, but still needs work before passing tests. Nice work! Although not sure if we should focus directly on sqlite4 (could be optional with a configure flag and a subtree [until stable], sqlite3 supported over depends). Also i personally would recommend to not implement it as a replacement, instead, support multiple backends (wrapper header / different wrapper implementations [leveldb / sqlite3 / sqlite4]). But – agreed – should not be the focus, but a nice additional flexibility if it not require much more work. And – in theory – multiple database back-ends would allow migration. Before investigating to deep, i think we need a dbwrapper bench tool that represents our needs and our style how we hit the database. Gavins recently added bench target / bench environment could be used for that. /jonas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJWKdlgAAoJECnUvLZBb1Ps7J8P/2L6215fd0rWGv5uvbLSnQvm hy1T2AaOfH5HXd2m95icaKYk+ugvQAL80Q/67YwZbPLsT4fErgegw8n75Z6nh/Pc OJ1EtAvD+Kc/Vm0Wcvt421HXwnm4f+j+8eoEvpG6DdC8gfIG+efhM7DljeNbPFrA ieNe7XrQ1EZ29lMzpQv0nx3bi6tUHOWuazk82B6vnK49MjH7nrUFipcc18nXbSpM ZKjQakXmfqIBG8QP9ZUYUlW4aoG0oaOoZnGjQA2LeXBJpIPLpE/WPg0XaXubC+No 232CJNtNyUOmjkb2qbep6vSYgqGJNy1HbCU5y3qooxJhFnKdo63CQsyJKSpL/ssi 0lWraNxjbacxsBn+63In3wEkj02orwm2zTO4I77wCrZmJgpvBFb9bZWTeL8DCYSG DTkZoKWEK74xvm+dNpEWXK9Lm1ltfyhPdaFeMoRDub4w2uuYlk3KuD8Vdy81HYJb sak8FbkiWw9xx2OP9+G56Arf9W6mnJ3YHJGrY4SXeRAfuNYwGFHIGt6I1JobuHy4 VWmcHuooz/Q9JLUu3Rr3HsEdXYgCmgmWuzTgWG+Hx92Y6XLWV5pOX+vFRO6J5fSm aTYPD4GsTM3FROXw5ezlXJ+2y1+ITzmrfm03fEDQvoIyH0TwBqBc5sMBBkma5DDR 0HUthPHWCD+AxbBPbRVa =CUMX -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-22 21:26 [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db Jeff Garzik ` (2 preceding siblings ...) 2015-10-23 6:53 ` Jonas Schnelli @ 2015-10-23 7:45 ` Lucas Betschart 2015-10-28 20:28 ` Sean Lynch 2015-10-23 10:30 ` Tom Zander 4 siblings, 1 reply; 36+ messages in thread From: Lucas Betschart @ 2015-10-23 7:45 UTC (permalink / raw) To: Jeff Garzik; +Cc: Bitcoin development mailing list [-- Attachment #1: Type: text/plain, Size: 971 bytes --] Facebook has a LevelDB fork which is maintained. It's called RocksDB and the API seems to be nearly the same as for LevelDB, thus maybe easy to replace: http://rocksdb.org/ https://github.com/facebook/rocksdb Although I don't know if we might have some negative effects for our use-case since RocksDB was optimized for big databases running on multiple cores. 2015-10-22 23:26 GMT+02:00 Jeff Garzik via bitcoin-dev < bitcoin-dev@lists•linuxfoundation.org>: > Here is the beginnings of an implementation to replace leveldb with > sqlite: https://github.com/jgarzik/bitcoin/tree/2015_sqlite > > It builds, but still needs work before passing tests. > > It was noted that leveldb is unmaintained, and this is part of researching > alternatives that are maintained and reliable. > > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > [-- Attachment #2: Type: text/html, Size: 1777 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-23 7:45 ` Lucas Betschart @ 2015-10-28 20:28 ` Sean Lynch 2015-10-28 21:11 ` Jeff Garzik 0 siblings, 1 reply; 36+ messages in thread From: Sean Lynch @ 2015-10-28 20:28 UTC (permalink / raw) To: Lucas Betschart, Jeff Garzik; +Cc: Bitcoin development mailing list [-- Attachment #1: Type: text/plain, Size: 610 bytes --] On Fri, Oct 23, 2015 at 1:23 AM Lucas Betschart via bitcoin-dev < bitcoin-dev@lists•linuxfoundation.org> wrote: > Facebook has a LevelDB fork which is maintained. > It's called RocksDB and the API seems to be nearly the same as for > LevelDB, thus maybe easy to replace: http://rocksdb.org/ > https://github.com/facebook/rocksdb > > Although I don't know if we might have some negative effects for our > use-case since RocksDB was optimized for big databases running on multiple > cores. > While RocksDB is pretty decent, note that it's optimized for flash. Not sure how well it will work on spinning disks. [-- Attachment #2: Type: text/html, Size: 1063 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-28 20:28 ` Sean Lynch @ 2015-10-28 21:11 ` Jeff Garzik 0 siblings, 0 replies; 36+ messages in thread From: Jeff Garzik @ 2015-10-28 21:11 UTC (permalink / raw) To: Sean Lynch; +Cc: Bitcoin development mailing list [-- Attachment #1: Type: text/plain, Size: 808 bytes --] On Wed, Oct 28, 2015 at 4:28 PM, Sean Lynch <seanl@literati•org> wrote: > On Fri, Oct 23, 2015 at 1:23 AM Lucas Betschart via bitcoin-dev < > bitcoin-dev@lists•linuxfoundation.org> wrote: > >> Facebook has a LevelDB fork which is maintained. >> It's called RocksDB and the API seems to be nearly the same as for >> LevelDB, thus maybe easy to replace: http://rocksdb.org/ >> https://github.com/facebook/rocksdb >> >> Although I don't know if we might have some negative effects for our >> use-case since RocksDB was optimized for big databases running on multiple >> cores. >> > > While RocksDB is pretty decent, note that it's optimized for flash. Not > sure how well it will work on spinning disks. > That's OK for our purposes. We have a huge database which already incentivized having zero seek time. [-- Attachment #2: Type: text/html, Size: 1673 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-22 21:26 [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db Jeff Garzik ` (3 preceding siblings ...) 2015-10-23 7:45 ` Lucas Betschart @ 2015-10-23 10:30 ` Tom Zander 2015-10-26 18:06 ` Douglas Roark 4 siblings, 1 reply; 36+ messages in thread From: Tom Zander @ 2015-10-23 10:30 UTC (permalink / raw) To: bitcoin-dev On Thursday 22 Oct 2015 17:26:42 Jeff Garzik via bitcoin-dev wrote: > It was noted that leveldb is unmaintained, and this is part of researching > alternatives that are maintained and reliable. Apart from it being unmaintained, any links to what are problems with levelDB? ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-23 10:30 ` Tom Zander @ 2015-10-26 18:06 ` Douglas Roark 2015-10-28 15:52 ` Tom Zander 2015-11-18 0:06 ` Jonathan Wilkins 0 siblings, 2 replies; 36+ messages in thread From: Douglas Roark @ 2015-10-26 18:06 UTC (permalink / raw) To: bitcoin-dev On 2015/10/23 03:30, Tom Zander via bitcoin-dev wrote: > On Thursday 22 Oct 2015 17:26:42 Jeff Garzik via bitcoin-dev wrote: >> It was noted that leveldb is unmaintained, and this is part of researching >> alternatives that are maintained and reliable. > > Apart from it being unmaintained, any links to what are problems with levelDB? While not exactly the most rigorous link, https://en.wikipedia.org/wiki/LevelDB#Bugs_and_Reliability seems like an okay place to start. One thing I can attest to is that, when Armory used LevelDB (0.8 - 0.92, IIRC), quite a few users had DB corruption issues, particularly on Windows. Even when a switch to LMDB occurred for 0.93, loads of complaints would come in from users whose LevelDB-based Core DBs would fail. I know that the guy who moved Armory over to LMDB would love to have more time in the day so that he could write a Core patch that does the same. It's a very sore spot for him. (FWIW, LMDB seems to work quite nicely, at least once you patch up the source a little bit. The latest version is also compatible with Core's cross-compiling scheme. I'd love to see it added to Core one day.) Doug ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-26 18:06 ` Douglas Roark @ 2015-10-28 15:52 ` Tom Zander 2015-11-18 0:06 ` Jonathan Wilkins 1 sibling, 0 replies; 36+ messages in thread From: Tom Zander @ 2015-10-28 15:52 UTC (permalink / raw) To: bitcoin-dev, Douglas Roark On Monday 26 Oct 2015 11:06:56 Douglas Roark via bitcoin-dev wrote: > While not exactly the most rigorous link, > https://en.wikipedia.org/wiki/LevelDB#Bugs_and_Reliability seems like an > okay place to start. Thanks for that link! Another Google open source product I'll avoid like the plague ;) ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-26 18:06 ` Douglas Roark 2015-10-28 15:52 ` Tom Zander @ 2015-11-18 0:06 ` Jonathan Wilkins 1 sibling, 0 replies; 36+ messages in thread From: Jonathan Wilkins @ 2015-11-18 0:06 UTC (permalink / raw) To: Douglas Roark; +Cc: bitcoin-dev [-- Attachment #1: Type: text/plain, Size: 1569 bytes --] Benchmarks for various DBs under discussion: http://symas.com/mdb/microbench/ On Mon, Oct 26, 2015 at 11:06 AM, Douglas Roark via bitcoin-dev < bitcoin-dev@lists•linuxfoundation.org> wrote: > On 2015/10/23 03:30, Tom Zander via bitcoin-dev wrote: > > On Thursday 22 Oct 2015 17:26:42 Jeff Garzik via bitcoin-dev wrote: > >> It was noted that leveldb is unmaintained, and this is part of > researching > >> alternatives that are maintained and reliable. > > > > Apart from it being unmaintained, any links to what are problems with > levelDB? > > While not exactly the most rigorous link, > https://en.wikipedia.org/wiki/LevelDB#Bugs_and_Reliability seems like an > okay place to start. One thing I can attest to is that, when Armory used > LevelDB (0.8 - 0.92, IIRC), quite a few users had DB corruption issues, > particularly on Windows. Even when a switch to LMDB occurred for 0.93, > loads of complaints would come in from users whose LevelDB-based Core > DBs would fail. I know that the guy who moved Armory over to LMDB would > love to have more time in the day so that he could write a Core patch > that does the same. It's a very sore spot for him. > > (FWIW, LMDB seems to work quite nicely, at least once you patch up the > source a little bit. The latest version is also compatible with Core's > cross-compiling scheme. I'd love to see it added to Core one day.) > > Doug > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > [-- Attachment #2: Type: text/html, Size: 2390 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db @ 2015-10-29 6:57 telemaco 2015-10-29 8:03 ` Luke Dashjr 2015-10-29 8:17 ` Gregory Maxwell 0 siblings, 2 replies; 36+ messages in thread From: telemaco @ 2015-10-29 6:57 UTC (permalink / raw) To: bitcoin-dev Why not allow two options: 1/ a default RocksDB/SQLite/LevelDB (whatever is decided) 2/ alternative provide instructions for connection to any other rdbms using odbc or jdbc. Why not allowing async disk writes or incredibly fast database systems if someone wants to have a node in a very fast datacenter or connected with their existing leveraged dataservers. It is the traditional approach to just use the open standard for database connectivity. Any person or any organization would just need to have one machine with their bitcoin node with a rdbms client installed (SAP Sybase client, or oracle client, or microsoft). The bitcoin node would just store their data using the odbc/jdbc protocol on ANY rdbms installed anywhere in their organization (other machine or the same). They would just need to issue a "create table" with a very simple table structure and they would benefit from async and indexes and using their already licensed, and configured system of their choosing, with bitcoin information being available to thousands of software packages and available aswell to thousands of programmers that work with rdbms and not just "RocksDB" or some obscure database system. Why not "outsource" totally that data management part to the already existing with decades of experience database world. People would be able to create incredibly easy bitcoin statistics/graphs/analisys with existing software packages (hey even excel or libreoffice like) or connect bitcoin data to their own sources and if so they chose analyze bitcoin data on a datawarehouse or any imaginable approach. Of course every transaction would be have to do through the bitcoin node and only the data management would be on rdbms side. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-29 6:57 telemaco @ 2015-10-29 8:03 ` Luke Dashjr 2015-10-30 3:04 ` Simon Liu 2015-10-29 8:17 ` Gregory Maxwell 1 sibling, 1 reply; 36+ messages in thread From: Luke Dashjr @ 2015-10-29 8:03 UTC (permalink / raw) To: telemaco; +Cc: bitcoin-dev On Thursday, October 29, 2015 6:57:39 AM telemaco via bitcoin-dev wrote: > Why not allow two options: > > 1/ a default RocksDB/SQLite/LevelDB (whatever is decided) > 2/ alternative provide instructions for connection to any other rdbms > using odbc or jdbc. I predict this would be a disaster. UTXO storage is CONSENSUS-CRITICAL code. Any divergence in implementation behaviour, including bugs AND bugfixes, may cause consensus failure. For this to have a reasonable *hope* of working, we need to choose one storage engine, and *will* need to maintain consensus- compatibility of it ourselves (since nobody else cares). Fixing LevelDB frankly seems like an easier task than switching to anything SQL-based, which would require a *lot* more *difficult-to-get-consensus- compatible* code that we are all (or at least mostly) very unfamiliar with. Research is fine, but let's be realistic about deployment. Luke ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-29 8:03 ` Luke Dashjr @ 2015-10-30 3:04 ` Simon Liu 2015-10-30 3:35 ` Gregory Maxwell 0 siblings, 1 reply; 36+ messages in thread From: Simon Liu @ 2015-10-30 3:04 UTC (permalink / raw) To: Luke Dashjr, telemaco; +Cc: bitcoin-dev Storage of UTXO data looks like an implementation detail and thus one would have thought that the choice of database would not increase the odds of consensus protocol failure. Btcd, a full node implementation written in Go, already provides a database interface which supports different backends: https://github.com/btcsuite/btcd/tree/master/database Given that UTXO storage is considered critical, it seems reasonable to let a node operator decide for themselves if they want data stored in LevelDB (which is not fully ACID compliant) or a database like Sqlite, Oracle, DB2 etc. If the storage requirements for UTXO data are fairly simple, consisting mainly of puts and gets, there is a decent argument that using a dedicated key-value store provides superior performance over a traditional SQL database. However, from a practical perspective, given that nodes operate on a range of different hardware and even a little Raspberry Pi can run a full node and keep up with the network, why not let those users with the resources to operate big iron databases do so? It would be a good feature to have. On 10/29/2015 01:03 AM, Luke Dashjr via bitcoin-dev wrote: > I predict this would be a disaster. UTXO storage is CONSENSUS-CRITICAL code. > Any divergence in implementation behaviour, including bugs AND bugfixes, may > cause consensus failure. For this to have a reasonable *hope* of working, we > need to choose one storage engine, and *will* need to maintain consensus- > compatibility of it ourselves (since nobody else cares). > > Fixing LevelDB frankly seems like an easier task than switching to anything > SQL-based, which would require a *lot* more *difficult-to-get-consensus- > compatible* code that we are all (or at least mostly) very unfamiliar with. > > Research is fine, but let's be realistic about deployment. > > Luke > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists•linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-30 3:04 ` Simon Liu @ 2015-10-30 3:35 ` Gregory Maxwell 2015-10-30 4:04 ` Peter R 0 siblings, 1 reply; 36+ messages in thread From: Gregory Maxwell @ 2015-10-30 3:35 UTC (permalink / raw) To: Simon Liu; +Cc: Bitcoin Dev, telemaco On Fri, Oct 30, 2015 at 3:04 AM, Simon Liu via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > Given that UTXO storage is considered critical, it seems reasonable to This sounds like a misunderstanding of what consensus criticial means. It does not mean that it must be right (though obviously that is preferable) but that it must be _consistent_, between all nodes. > full node and keep up with the network, why not let those users with the > resources to operate big iron databases do so? It would be a good > feature to have. Because it provides no value, the data is opaque and propritarily encoded with a compression function which we may change from version to version, and because many of these alternatives are enormously slow; enough that they present problems with falling behind the network even on high performance hardware. Moreover, additional functional which will not be sufficiently used will not adequately maintained and result in increased maintains costs and more bugs. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-30 3:35 ` Gregory Maxwell @ 2015-10-30 4:04 ` Peter R 2015-10-30 4:28 ` Gregory Maxwell 0 siblings, 1 reply; 36+ messages in thread From: Peter R @ 2015-10-30 4:04 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Bitcoin Dev, telemaco > On Oct 29, 2015, at 8:35 PM, Gregory Maxwell via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > > On Fri, Oct 30, 2015 at 3:04 AM, Simon Liu via bitcoin-dev > <bitcoin-dev@lists•linuxfoundation.org> wrote: >> Given that UTXO storage is considered critical, it seems reasonable to > > This sounds like a misunderstanding of what consensus criticial means. > It does not mean that it must be right (though obviously that is > preferable) but that it must be _consistent_, between all nodes. Can you give a specific example of how nodes that used different database technologies might determine different answers to whether a given transaction is valid or invalid? I’m not a database expert, but to me it would seem that if all the unspent outputs can be found in the database, and if the relevant information about each output can be retrieved without corruption, then that’s all that really matters as far as the database is concerned. Let’s use an unspent pay-to-pubkey-hash output as an example: Alice spends this to Bob (she signs it properly), the TX propagates across the network and…then what? Do some nodes disagree on whether or not the TX is valid? What exactly would they disagree on? Are you suggesting that a database bug would cause some nodes to think the output was actually already spent, while others can correctly see that it’s unspent? Or maybe some nodes think the output doesn’t exist while others do? Or are you suggesting that the details about this output might be retrieved with errors from certain databases but correctly from others? I’d like a concrete example to help me understand why more than one implementation of something like the UTXO database would be unreasonable. Peter ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-30 4:04 ` Peter R @ 2015-10-30 4:28 ` Gregory Maxwell 2015-11-15 1:02 ` Peter R 0 siblings, 1 reply; 36+ messages in thread From: Gregory Maxwell @ 2015-10-30 4:28 UTC (permalink / raw) To: Peter R; +Cc: Bitcoin Dev, telemaco On Fri, Oct 30, 2015 at 4:04 AM, Peter R <peter_r@gmx•com> wrote: > Can you give a specific example of how nodes that used different database technologies might determine different answers to whether a given transaction is valid or invalid? I’m not a database expert, but to me it would seem that if all the unspent outputs can be found in the database, and if the relevant information about each output can be retrieved without corruption, then that’s all that really matters as far as the database is concerned. If you add to those set of assumptions the handling of write ordering is the same (e.g. multiple updates in an change end up with the same entry surviving) and read/write interleave returning the same results then it wouldn't. But databases sometimes have errors which cause them to fail to return records, or to return stale data. And if those exist consistency must be maintained; and "fixing" the bug can cause a divergence in consensus state that could open users up to theft. Case in point, prior to leveldb's use in Bitcoin Core it had a bug that, under rare conditions, could cause it to consistently return not found on records that were really there (I'm running from memory so I don't recall the specific cause). Leveldb fixed this serious bug in a minor update. But deploying a fix like this in an uncontrolled manner in the bitcoin network would potentially cause a fork in the consensus state; so any such fix would need to be rolled out in an orderly manner. > I’d like a concrete example to help me understand why more than one implementation of something like the UTXO database would be unreasonable. It's not unreasonable, but great care is required around the specifics. Bitcoin consensus implements a mathematical function that defines the operation of the system and above all else all systems must agree (or else the state can diverge and permit double-spends); if you could prove that a component behaves identically under all inputs to another function then it can be replaced without concern but this is something that cannot be done generally for all software, and proving equivalence even in special cases it is an open area of research. The case where the software itself is identical or nearly so is much easier to gain confidence in the equivalence of a change through testing and review. With that cost in mind one must then consider the other side of the equation-- utxo database is an opaque compressed representation, several of the posts here have been about desirability of blockchain analysis interfaces, and I agree they're sometimes desirable but access to the consensus utxo database is not helpful for that. Similarly, other things suggested are so phenomenally slow that it's unlikely that a node would catch up and stay synced even on powerful hardware. Regardless, in Bitcoin core the storage engine for this is fully internally abstracted and so it is relatively straight forward for someone to drop something else in to experiment with; whatever the motivation. I think people are falling into a trap of thinking "It's a <database>, I know a <black box> for that!"; but the application and needs are very specialized here; no less than, say-- the table of pre-computed EC points used for signing in the ECDSA application. It just so happens that on the back of the very bitcoin specific cryptographic consensus algorithim there was a slot where a pre-existing high performance key-value store fit; and so we're using one and saving ourselves some effort. If, in the future, Bitcoin Core adopts a merkelized commitment for the UTXO it would probably need to stop using any off-the-shelf key value store entirely, in order to avoid a 20+ fold write inflation from updating hash tree paths (And Bram Cohen has been working on just such a thing, in fact). ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-30 4:28 ` Gregory Maxwell @ 2015-11-15 1:02 ` Peter R 2015-11-15 1:08 ` Gregory Maxwell 2015-11-15 3:04 ` Luke Dashjr 0 siblings, 2 replies; 36+ messages in thread From: Peter R @ 2015-11-15 1:02 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Bitcoin Dev, telemaco Hi Greg, Like you said, the issue with using more than one database technology is not that one node would prove that Block X is valid while the another node proves that Block X is NOT valid. Instead, the problem is that one node might say “valid” while the other node says “I don’t know.” It is often said that what caused the Level DB fork was that the old version determined that the triggering block was invalid while the new version determined the opposite. This interpretation of the fork event has lead to the “bug-for-bug”-compatibility fear regarding multiple implementations of the protocol (or multiple database technologies). In reality, this fear is largely unfounded. The old version ran out of LDB locks and couldn’t execute properly. If the software was written with the philosophy that tracking consensus was more important than bug-for-bug compatibility, it would have returned an exception rather than “invalid.” At that point, the software would have checked what decision the rest of the network came to about the block in question. My node would have forked to the longest chain, automatically assuming that the questionable block was valid; your node may have made a different decision (it’s a question of ideology at that point). A group of us have been exploring this “meta-cognition” idea with Bitcoin Unlimited. For example, Bitcoin Unlimited can be (optionally) made to automatically fork to the longest chain if it “gets stuck” and can neither prove that a block is valid nor that the block is invalid. Similarly, Bitcoin Unlimited can be (optionally) made to automatically fork to a chain that contains a block marginally bigger than its block size limit—once that block is buried sufficiently in what has emerged as the longest persistent chain. Thinking from this perspective might go along way towards decentralizing development, the emergence of multiple competing implementations of the protocol, and true “bottom up” governance. Best regards, Peter > On Oct 29, 2015, at 9:28 PM, Gregory Maxwell <gmaxwell@gmail•com> wrote: > > On Fri, Oct 30, 2015 at 4:04 AM, Peter R <peter_r@gmx•com> wrote: >> Can you give a specific example of how nodes that used different database technologies might determine different answers to whether a given transaction is valid or invalid? I’m not a database expert, but to me it would seem that if all the unspent outputs can be found in the database, and if the relevant information about each output can be retrieved without corruption, then that’s all that really matters as far as the database is concerned. > > If you add to those set of assumptions the handling of write ordering > is the same (e.g. multiple updates in an change end up with the same > entry surviving) and read/write interleave returning the same results > then it wouldn't. > > But databases sometimes have errors which cause them to fail to return > records, or to return stale data. And if those exist consistency must > be maintained; and "fixing" the bug can cause a divergence in > consensus state that could open users up to theft. > > Case in point, prior to leveldb's use in Bitcoin Core it had a bug > that, under rare conditions, could cause it to consistently return not > found on records that were really there (I'm running from memory so I > don't recall the specific cause). Leveldb fixed this serious bug in a > minor update. But deploying a fix like this in an uncontrolled manner > in the bitcoin network would potentially cause a fork in the consensus > state; so any such fix would need to be rolled out in an orderly > manner. > >> I’d like a concrete example to help me understand why more than one implementation of something like the UTXO database would be unreasonable. > > It's not unreasonable, but great care is required around the specifics. > > Bitcoin consensus implements a mathematical function that defines the > operation of the system and above all else all systems must agree (or > else the state can diverge and permit double-spends); if you could > prove that a component behaves identically under all inputs to another > function then it can be replaced without concern but this is something > that cannot be done generally for all software, and proving > equivalence even in special cases it is an open area of research. The > case where the software itself is identical or nearly so is much > easier to gain confidence in the equivalence of a change through > testing and review. > > With that cost in mind one must then consider the other side of the > equation-- utxo database is an opaque compressed representation, > several of the posts here have been about desirability of blockchain > analysis interfaces, and I agree they're sometimes desirable but > access to the consensus utxo database is not helpful for that. > Similarly, other things suggested are so phenomenally slow that it's > unlikely that a node would catch up and stay synced even on powerful > hardware. Regardless, in Bitcoin core the storage engine for this is > fully internally abstracted and so it is relatively straight forward > for someone to drop something else in to experiment with; whatever the > motivation. > > I think people are falling into a trap of thinking "It's a <database>, > I know a <black box> for that!"; but the application and needs are > very specialized here; no less than, say-- the table of pre-computed > EC points used for signing in the ECDSA application. It just so > happens that on the back of the very bitcoin specific cryptographic > consensus algorithim there was a slot where a pre-existing high > performance key-value store fit; and so we're using one and saving > ourselves some effort. If, in the future, Bitcoin Core adopts a > merkelized commitment for the UTXO it would probably need to stop > using any off-the-shelf key value store entirely, in order to avoid a > 20+ fold write inflation from updating hash tree paths (And Bram Cohen > has been working on just such a thing, in fact). ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 1:02 ` Peter R @ 2015-11-15 1:08 ` Gregory Maxwell 2015-11-15 1:45 ` Peter R 2015-11-15 3:04 ` Luke Dashjr 1 sibling, 1 reply; 36+ messages in thread From: Gregory Maxwell @ 2015-11-15 1:08 UTC (permalink / raw) To: Peter R; +Cc: Bitcoin Dev, telemaco On Sun, Nov 15, 2015 at 1:02 AM, Peter R <peter_r@gmx•com> wrote: > Hi Greg, > > Like you said, the issue with using more than one database technology is not that one node would prove that Block X is valid while the another node proves that Block X is NOT valid. Instead, the problem is that one node might say “valid” while the other node says “I don’t know.” Sometimes errors are such that you can catch them (if you're super vigilant and know an error is even possible in that case)-- and indeed, in that case you can get a "I don't know, something is wrong.", other times errors are undetectable. > In reality, this fear is largely unfounded. I cited an issue in leveldb (before we used it) where it would silently fail to return records. > If the software was written with the philosophy that tracking consensus was more important than bug-for-bug compatibility, it would have returned an exception rather than “invalid.” That invariant requires the software to be completely free of errors that would violate it. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 1:08 ` Gregory Maxwell @ 2015-11-15 1:45 ` Peter R 2015-11-15 2:10 ` Gregory Maxwell 0 siblings, 1 reply; 36+ messages in thread From: Peter R @ 2015-11-15 1:45 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Bitcoin Dev, telemaco > On Nov 14, 2015, at 5:08 PM, Gregory Maxwell <gmaxwell@gmail•com> wrote: > > On Sun, Nov 15, 2015 at 1:02 AM, Peter R <peter_r@gmx•com> wrote: >> Hi Greg, >> >> Like you said, the issue with using more than one database technology is not that one node would prove that Block X is valid while the another node proves that Block X is NOT valid. Instead, the problem is that one node might say “valid” while the other node says “I don’t know.” > > Sometimes errors are such that you can catch them (if you're super > vigilant and know an error is even possible in that case)-- and > indeed, in that case you can get a "I don't know, something is > wrong.", other times errors are undetectable. Agreed. There are two cases to consider: Type 1. One implementation says “yes” or “no,” while the other says “I don’t know”, and Type 2. One implementation says “yes” and the other says “no,” because of a bug. My previous email described how Type 1 consensus failures can be safely dealt with. These include many kinds of database exceptions (e.g., the LevelDB fork at block #225,430), or consensus mismatches regarding the max size of a block. Type 2 consensus failures are more severe but also less likely (I’m not aware of a Type 2 consensus failure besides the 92 million bitcoin bug from August 2010). If Core was to accept a rogue TX that created another 92 million bitcoins, I think it would be a good thing if the other implementations forked away from it (we don’t want bug-for-bug compatibility here). This once again reveals the benefits of multiple competing implementations. Sincerely, Peter ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 1:45 ` Peter R @ 2015-11-15 2:10 ` Gregory Maxwell 2015-11-15 2:58 ` Peter R 0 siblings, 1 reply; 36+ messages in thread From: Gregory Maxwell @ 2015-11-15 2:10 UTC (permalink / raw) To: Peter R; +Cc: Bitcoin Dev, telemaco On Sun, Nov 15, 2015 at 1:45 AM, Peter R <peter_r@gmx•com> wrote: > My previous email described how Type 1 consensus failures can be safely dealt with. These include many kinds of database exceptions (e.g., the LevelDB fork at block #225,430), or consensus mismatches regarding the max size of a block. The size of a block is not a type 1 failure. There is a clear, known, unambiguous, and easily measured rule in the system that a block is not permitted to have a size over a threshold. A block that violates this threshold is invalid. The only way I can see to classify that as a type 1 failure is to call the violation of any known system rule a type 1 failure. "Spends a coin that was already spent" "provides a signature that doesn't verify with the pubkey" "creates bitcoins out of thin air" -- these are not logically different than "if size>x return false". > Type 2 consensus failures are more severe but also less likely (I’m not aware of a Type 2 consensus failure besides the 92 million bitcoin bug from August 2010). If Core was to accept a rogue TX that created another 92 million bitcoins, I think it would be a good thing if the other implementations forked away from it (we don’t want bug-for-bug compatibility here). The only consensus consistency error we've ever that I would have classified as potentially type 1 had has been the BDB locks issue. Every other one, including the most recently fixed one (eliminated by BIP66) was a type 2, by your description. They are _much_ more common; because if the author understood the possible cases well enough to create an "I don't know" case, they could have gone one step further and fully defined it in a consistent way. The fact that an inconsistency was possible was due to a lack of complete knowledge. Even in the BDB locks case, I don't know that the type 1 distinction completely applies, a lower layer piece of software threw an error that higher layer software didn't know was possible, and so it interpreted "I don't know" as "no"-- it's not that it chose to treat I don't know as no, its that it wasn't authored with the possibility of I don't know in mind, and that exceptions were used to handle general failures (which should have been treated as no). Once you step back to the boundary of the calling code (much less the whole application) the "I don't know" doesn't exist anymore; there. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 2:10 ` Gregory Maxwell @ 2015-11-15 2:58 ` Peter R 2015-11-15 3:30 ` Gregory Maxwell 2015-11-16 1:52 ` Rusty Russell 0 siblings, 2 replies; 36+ messages in thread From: Peter R @ 2015-11-15 2:58 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Bitcoin Dev, telemaco > On Nov 14, 2015, at 6:10 PM, Gregory Maxwell <gmaxwell@gmail•com> wrote: > > On Sun, Nov 15, 2015 at 1:45 AM, Peter R <peter_r@gmx•com> wrote: >> My previous email described how Type 1 consensus failures can be safely dealt with. These include many kinds of database exceptions (e.g., the LevelDB fork at block #225,430), or consensus mismatches regarding the max size of a block. > > The size of a block is not a type 1 failure. There is a clear, known, > unambiguous, and easily measured rule in the system that a block is > not permitted to have a size over a threshold. A block that violates > this threshold is invalid. You are looking at the problem from a “top down” governance perspective assuming you know what code is actually being run and what rules the market wants. The reality is that you can only make estimates about these two things. As Bitcoin evolves away from the current development monoculture, rules such as the max block size may no longer be perfectly clear. However, we can prove that the following results will continue to hold: 1. A node with a block size limit greater than the hash-power weighted median will always follow the longest chain. 2. An excessive (e.g., greater than 1 MB) block will be accepted into the longest chain if it is smaller than the hash-power weighted median block size limit. Already today, different nodes have different block size limits. There are even some miners today that will attempt to build upon blocks larger than 1 MB if anyone dares to create one. At some point, the majority of the hash power will support something larger than 1 MB. When that first bigger block comes and gets buried in the longest chain, hold-out nodes (e.g., MP says he will never increase his limit) will have to make a choice: fight consensus or track consensus! I know that I would want my node to give up on its block size limit and track consensus. You may decide to make a different choice. You said that "a block that violates this [block size limit] threshold is invalid.” I agree. If the nodes and miners rejected the block as invalid then it would not persist as the longest chain. If the nodes and miners accepted the block and continued to build on top of it, then that chain would be Bitcoin (whether you personally agree of not). Bitcoin is ultimately a creature of the market, governed by the code people freely choose to run. Consensus is then an emergent property, objectively represented by the longest persistent chain. Proof-of-work both enforces and defines the rules of the network. > The only way I can see to classify that > as a type 1 failure is to call the violation of any known system rule > a type 1 failure. "Spends a coin that was already spent" "provides a > signature that doesn't verify with the pubkey" "creates bitcoins out > of thin air" -- these are not logically different than "if size>x > return false”. I think you’re being intentionally obtuse here: accepting a block composed entirely of valid transactions that is 1.1 MB is entirely different than accepting a TX that creates a ten thousand bitcoins out of thin air. The market would love the former but abhor the later. I believe you can recognize the difference. >> Type 2 consensus failures are more severe but also less likely (I’m not aware of a Type 2 consensus failure besides the 92 million bitcoin bug from August 2010). If Core was to accept a rogue TX that created another 92 million bitcoins, I think it would be a good thing if the other implementations forked away from it (we don’t want bug-for-bug compatibility here). > > The only consensus consistency error we've ever that I would have > classified as potentially type 1 had has been the BDB locks issue. Thank you for conceding on that point. > Every other one, including the most recently fixed one (eliminated by > BIP66) was a type 2, by your description. They are _much_ more common; > because if the author understood the possible cases well enough to > create an "I don't know" case, they could have gone one step further > and fully defined it in a consistent way. The fact that an > inconsistency was possible was due to a lack of complete knowledge. > > Even in the BDB locks case, I don't know that the type 1 distinction > completely applies, a lower layer piece of software threw an error > that higher layer software didn't know was possible, and so it > interpreted "I don't know" as "no"-- it's not that it chose to treat I > don't know as no, its that it wasn't authored with the possibility of > I don't know in mind, and that exceptions were used to handle general > failures (which should have been treated as no). Once you step back to > the boundary of the calling code (much less the whole application) the > "I don't know" doesn't exist anymore; there. Please don’t take my comments and observations as criticisms. I think the Core Dev team has done excellent work! What I am saying instead is that as we move forward—as development becomes decentralized and multiple protocol implementations emerge—development philosophies will change. Tracking consensus and the will of the market will be most important. Personally, I hope to see design philosophies that support “bottom up” governance instead of the current “top down” model. Best regards, Peter ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 2:58 ` Peter R @ 2015-11-15 3:30 ` Gregory Maxwell 2015-11-15 4:10 ` Peter R 2015-11-16 1:52 ` Rusty Russell 1 sibling, 1 reply; 36+ messages in thread From: Gregory Maxwell @ 2015-11-15 3:30 UTC (permalink / raw) To: Peter R; +Cc: Bitcoin Dev, telemaco On Sun, Nov 15, 2015 at 2:58 AM, Peter R <peter_r@gmx•com> wrote: > I think you’re being intentionally obtuse here: accepting a block composed entirely of valid transactions that is 1.1 MB is entirely different than accepting a TX that creates a ten thousand bitcoins out of thin air. The market would love the former but abhor the later. I believe you can recognize the difference. It is not technically distinct--today; politically-- perhaps, but-- sorry, no element of your prior message indicated that you were interested in discussing politics rather than technology; on a mailing list much more strongly scoped for the latter; I hope you can excuse me for missing your intention prior to your most recent post. That said, I believe you are privileging your own political preferences in seeing the one rule of the bitcoin system as categorically distinct even politically. No law of nature leaves the other criteria I specified less politically negotiable, and we can see concrete examples all around us -- the notion that funds can be confiscated via external authority (spending without the owners signature) is a more or less universal property of other modern systems of money, that economic controls out to exist to regulate the supply of money for the good of an economy is another widely deployed political perspective. You, yourself, recently published a work on the stable self regulation of block sizes based on mining incentives that took as its starting premise a bitcoin that was forever inflationary. Certainly things differ in degrees, but this is not the mailing list to debate the details of political inertia. > Thank you for conceding on that point. You're welcome, but I would have preferred that you instead of your thanks you would have responded in kind and acknowledged my correction that other consensus inconsistencies discovered in implementations thus far (none, that I'm aware of) could be classified as "maybe"; and in doing so retained a semblance of a connection to a the technical purposes of this mailing list. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 3:30 ` Gregory Maxwell @ 2015-11-15 4:10 ` Peter R 2015-11-15 10:12 ` Jorge Timón 0 siblings, 1 reply; 36+ messages in thread From: Peter R @ 2015-11-15 4:10 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Bitcoin Dev, telemaco Hi Greg, >> Thank you for conceding on that point. > > You're welcome, but I would have preferred that you instead of your > thanks you would have responded in kind and acknowledged my correction > that other consensus inconsistencies discovered in implementations > thus far (none, that I'm aware of) could be classified as "maybe"; and > in doing so retained a semblance of a connection to a the technical > purposes of this mailing list. I apologize for that, Greg. I have not read enough on the issues you brought up to comment intelligibly. I should have conceded that you could very well be correct that those were Type 2 consensus failures. >> I think you’re being intentionally obtuse here: accepting a block composed entirely of valid transactions that is 1.1 MB is entirely different than accepting a TX that creates a ten thousand bitcoins out of thin air. The market would love the former but abhor the later. I believe you can recognize the difference. > > It is not technically distinct--today; politically-- perhaps, but-- > sorry, no element of your prior message indicated that you were > interested in discussing politics rather than technology; on a mailing > list much more strongly scoped for the latter; I hope you can excuse > me for missing your intention prior to your most recent post. The difference between a 1.1 MB block full of valid transactions and an invalid TX that creates 10,000 BTC out of thin air is *not* a matter of “politics.” If people could freely award themselves coins, then Bitcoin would not be money. It’s like saying that “technically” there’s no difference between picking up a penny from the sidewalk and holding up a bank teller at gunpoint. Ask the average person: there is more than a “political” difference between creating coins out of thin air and increasing the block size limit. > That said, I believe you are privileging your own political > preferences in seeing the one rule of the bitcoin system as > categorically distinct even politically. What rules does Bitcoin obey? What is Bitcoin’s nature? This brings us to the age-old debate between Rationalism versus Empiricism. Rationalism holds that some propositions are known to be true by intuition alone and that others are knowable by being deduced from intuited propositions. The Rationalist may hold the view that Bitcoin has a 21-million coin limit or a 1 MB block size limit, based on deductive reasoning from the rules enforced by the Bitcoin Core source code. Such a Rationalists might believe that the code represents some immutable truth and then his understanding of Bitcoin follows from axiomatic deductions from that premise. The Empiricist rejects the Rationalist’s intuition and deduction, believing instead that knowledge is necessarily a posteriori, dependent upon observation and sense experience. The Empiricist questions the notion that Bitcoin has a 21-million coin limit, instead observing that its money supply grew by 50 BTC per block for the first 210,000 and then 25 BTC per block ever since. The Empiricist rejects the idea that Bitcoin has any sort of block size limit, having observed previous empirical limits collapse in the face of increased demand. I am not convinced that Bitcoin even *has* a block size limit, let alone that it can enforce one against the invisible hand of the market. > No law of nature leaves the > other criteria I specified less politically negotiable, and we can see > concrete examples all around us -- the notion that funds can be > confiscated via external authority (spending without the owners > signature) is a more or less universal property of other modern > systems of money, that economic controls out to exist to regulate the > supply of money for the good of an economy is another widely deployed > political perspective. You, yourself, recently published a work on the > stable self regulation of block sizes based on mining incentives that > took as its starting premise a bitcoin that was forever inflationary. > Certainly things differ in degrees, but this is not the mailing list > to debate the details of political inertia. You were the one who just brought up politics, Greg. Not I. Best regards, Peter ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 4:10 ` Peter R @ 2015-11-15 10:12 ` Jorge Timón 2015-11-15 11:28 ` Jorge Timón 2015-11-15 17:06 ` Peter R 0 siblings, 2 replies; 36+ messages in thread From: Jorge Timón @ 2015-11-15 10:12 UTC (permalink / raw) To: Peter R; +Cc: Bitcoin Dev, Gregory Maxwell, telemaco [-- Attachment #1: Type: text/plain, Size: 2124 bytes --] On Nov 15, 2015 5:10 AM, "Peter R via bitcoin-dev" < bitcoin-dev@lists•linuxfoundation.org> wrote: > > What rules does Bitcoin obey? Thanks to the worl of many people, part of the consensus rules are finally encapsulated in the libbitcoinconsensus library. I'm currently writing a document to complete the encapsulation of the specification of the consensus rules. > I am not convinced that Bitcoin even *has* a block size limit, let alone that it can enforce one against the invisible hand of the market. You keep insisting that some consensus rules are not consensus rules while others "are clearly a very different thing". What technical difference is there between the rule that impedes me from creating transactions bigger than X and the rules that prevent me frm creatin new coins (not as a miner, as a regular user in a transaction with more coins in the outputs than in the inputs)? What about property enforcement? If the invisible hand of the market is what decides consensus rules instead of their (still incomple) specification (aka libconsensus), then the market could decide to stop enforcing ownership. Will you still think that Bitcoin is a useful system when/if you empirically observe the invisible hand of the market taking coins out of your pocket? You also keep assuming that somehow it is a universal law that users must eventually converge under the most-work chain. People follow the most-work VALID chain, but if they consciously decide to implement different rules (different definitions of "valid block") then their chains can diverge, and once they do they won't converge again (unless/until one group decides to implement the rules of the other exactly again), just like when the implementation of the rules diverge in a unintentional consensus fork. But in this case they could decide to never implement the same rules. See bip99 and specially the "schism hardforks" section for more details. > You were the one who just brought up politics, Greg. Not I. Please, read the thread again. I think it is pretty clear that you did. Nothing wrong with that, just move it to the discussion ml. [-- Attachment #2: Type: text/html, Size: 2397 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 10:12 ` Jorge Timón @ 2015-11-15 11:28 ` Jorge Timón 2015-11-15 15:48 ` Peter R 2015-11-15 17:06 ` Peter R 1 sibling, 1 reply; 36+ messages in thread From: Jorge Timón @ 2015-11-15 11:28 UTC (permalink / raw) To: Peter R; +Cc: Bitcoin Dev, Gregory Maxwell, telemaco Going back on topic, I believe libconsensus shouldn't depend on any particular database because assuming it will continue to be stateless (the current libbitcoinconsensus is stateless) end therefore has no storage. I know some people disagree in various degrees. At the same time, the parts of the consensus rules verification that depends on storage has not been encapsulated out to libbitcoinconsensus yet, and I agree that changing the database is unnecessarily risky at this point. Even when the consensus rules are encapsulated, that doesn't mean that Bitcoin Core should be DB agnostic or that we can guarantee that it will follow the longest valid chain with databases that have not been tested. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 11:28 ` Jorge Timón @ 2015-11-15 15:48 ` Peter R 0 siblings, 0 replies; 36+ messages in thread From: Peter R @ 2015-11-15 15:48 UTC (permalink / raw) To: Jorge Timón; +Cc: Bitcoin Dev, Gregory Maxwell, telemaco > On Nov 15, 2015, at 3:28 AM, Jorge Timón <jtimon@jtimon•cc> wrote: > > Going back on topic, I believe libconsensus shouldn't depend on any > particular database because assuming it will continue to be stateless > (the current libbitcoinconsensus is stateless) end therefore has no > storage. Agreed. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 10:12 ` Jorge Timón 2015-11-15 11:28 ` Jorge Timón @ 2015-11-15 17:06 ` Peter R 2015-11-17 13:54 ` Tamas Blummer 1 sibling, 1 reply; 36+ messages in thread From: Peter R @ 2015-11-15 17:06 UTC (permalink / raw) To: Jorge Timón; +Cc: Bitcoin Dev, Gregory Maxwell, telemaco [-- Attachment #1: Type: text/plain, Size: 3838 bytes --] Hello Jorge: > > What rules does Bitcoin obey? > > Thanks to the worl of many people, part of the consensus rules are finally encapsulated in the libbitcoinconsensus library. I'm currently writing a document to complete the encapsulation of the specification of the consensus rules. > I applaud your work on the consensus library. I think it an important step to encouraging multiple competing implementations of the protocol. > > I am not convinced that Bitcoin even *has* a block size limit, let alone that it can enforce one against the invisible hand of the market. > > You keep insisting that some consensus rules are not consensus rules while others "are clearly a very different thing". What technical difference is there between the rule that impedes me from creating transactions bigger than X and the rules that prevent me frm creatin new coins (not as a miner, as a regular user in a transaction with more coins in the outputs than in the inputs)? > What technical difference is there between a cat and a dog? They both have four legs and a furry coat. I think you’re using the term “technical difference” to mean something very specific. Perhaps you could clarify exactly how you are defining that term because to me it is crystal clear that creating coins out of thin air is very different than accepting a block 1.1 MB in size and full of valid TXs. There are many technical differences between the two. For example, technically the first allows coins to be created randomly while the second doesn’t. > What about property enforcement? If the invisible hand of the market is what decides consensus rules instead of their (still incomple) specification (aka libconsensus), then the market could decide to stop enforcing ownership. > Correct. Bitcoin is an experiment and could still fail (e.g., the network could allow people to move coins without valid signatures). For Bitcoin to be viable, the network of miners and node operators being net-econo-rational actually is probably a core axiom that must be accepted. > You also keep assuming that somehow it is a universal law that users must eventually converge under the most-work chain. People follow the most-work VALID chain, but if they consciously decide to implement different rules (different definitions of "valid block") then their chains can diverge, and once they do they won't converge again (unless/until one group decides to implement the rules of the other exactly again) > It is fact that two competing forks can persist for at least a short amount of time—we saw this a few years ago with the LevelDB bug and again this summer with the SPV mining incident. In both cases, there was tremendous pressure to converge back to a single chain. Could two chains persist indefinitely? I don’t know. No one knows. My gut feeling is that since users would have coins on both sides of the fork, there would be a fork arbitrage event (a “forkbitrage”) where speculators would sell the coins on the side they predict to lose in exchange for additional coins on the side they expect to win. This could actually be facilitated by exchanges once the fork event is credible and before the fork actually occurs, or even in a futures market somehow. I suspect that the forkbitrage would create an unstable equilibrium where coins on one side quickly devalue. Miners would then abandon that side in favour of the other, killing the fork because difficulty would be too high to find new blocks. Anyways, I think even *this* would be highly unlikely. I suspect nodes and miners would get inline with consensus as soon as the fork event was credible. Cryptocurrency is a new area of interdisciplinary research. We are all learning together and no one knows how things will unfold. Best regards, Peter [-- Attachment #2: Type: text/html, Size: 4698 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 17:06 ` Peter R @ 2015-11-17 13:54 ` Tamas Blummer 2015-11-17 15:24 ` Tom Harding 0 siblings, 1 reply; 36+ messages in thread From: Tamas Blummer @ 2015-11-17 13:54 UTC (permalink / raw) To: Peter R, Peter R via bitcoin-dev; +Cc: Gregory Maxwell, telemaco [-- Attachment #1: Type: text/plain, Size: 718 bytes --] Isolating storage from the rest of consensus code is technically desirable, but implementations using different storage will be unlikely bug-for-bug compatible, hence able to split the network. Such split was disastrous on the network level if partitions were of comparable magnitude - as was the case in the March 2013 fork between versions of Bitcoin Core. This means high level implementation diversity was great, provided we would get there without blowing up the network on the way from monoculture to true decentralization of code. Libconsensus is immensely valuable to get diversity, as it makes alternate implementations bug-for-bug compatible for a big part of the consensus code. Tamas Blummer [-- Attachment #2: Message signed with OpenPGP using GPGMail --] [-- Type: application/pgp-signature, Size: 496 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-17 13:54 ` Tamas Blummer @ 2015-11-17 15:24 ` Tom Harding 2015-11-17 22:17 ` telemaco 0 siblings, 1 reply; 36+ messages in thread From: Tom Harding @ 2015-11-17 15:24 UTC (permalink / raw) To: Tamas Blummer; +Cc: Bitcoin Dev, Gregory Maxwell, telemaco [-- Attachment #1: Type: text/plain, Size: 420 bytes --] On Nov 17, 2015 5:54 AM, "Tamas Blummer via bitcoin-dev" < bitcoin-dev@lists•linuxfoundation.org> wrote: > > Isolating storage from the rest of consensus code is technically desirable, but implementations using different storage will be unlikely bug-for-bug compatible, > hence able to split the network. The problem with unknown bugs is you don't know how serious they are. A serious bug could itself be devastating. [-- Attachment #2: Type: text/html, Size: 562 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-17 15:24 ` Tom Harding @ 2015-11-17 22:17 ` telemaco 2015-11-20 14:15 ` Jorge Timón 0 siblings, 1 reply; 36+ messages in thread From: telemaco @ 2015-11-17 22:17 UTC (permalink / raw) To: Tom Harding, Tamas Blummer; +Cc: Bitcoin Dev, Gregory Maxwell [-- Attachment #1: Type: text/plain, Size: 3521 bytes --] Shouldn't a odbc jdbc jconnect or equivalent be totally transparent for the consensus code? I mean, the client would write or store the data communicating to the driver provided by the vendor. Using the schema bitcoin suggests adapted to many different vendors (one table schema for Oracle, other for mysql, etc with their slight syntax particularities), installed in the machine with the node and from that communication to the driver the storage would be totally controlled by the third party rdbms. Regarding bugs or risk of fork, does not have actual client any defense against someone forking core and slightly changing the actual database used maybe wrongly and creating a fork by themselves? Does the client have any way to verify that what is stored is correct? Maybe inserting a column with a hash of what is stored in each row and another column with a incremental row by row hash composed by the hash of each row and the previous column one., so any tampering in a previous row can be verified up to where is not consistent. I just imagine what would be for people to be able to access easily (with the thousands of software packages already bought and licensed by ALL companies in the world that already use open standard connectivity or equivalents)., the bitcoin blockchain. SUBSCRIPTION: for a couple decades replication servers have allowed a publish/subscription model using replication agents. If I am a guy working on a lever in the warehouse with my pda I do not need on my pda all the company info or maybe all the blockchain. If a company., that has already licensed a rdbms package with dozens of related software packages needs one guy to suscribe to something on the bitcoin blockchain, he can either use one of the purchased methods in their company and access the company database that holds blockchain data or hire a rare bitcoin developer that will create a interfaz bitcoin for a specific need up to the millions of needs out there. PUBLISHING Maybe even to have a publishing daemon that would allow those companies and their software packages to write things in the bitcoin blockchain provided of couse that they fund the agent with a small bitcoin amount to send transactions and they comply with the database constraint of being the owners of the private key. The publishing agent would check for changes every X minutes on that specific address in the db and if funded it would publish "send" the transaction through the bitcoin client. People would be able to publish info on the decentralized ledger from 90% of enterprise software packages.,paying ofc and with the small delay of the publishing agent checking for changes. In fact the db would allow publishing info while the publishing agent could just take its time publishing at its own rate like a slow write cache. In any case shouldn't even actual consensus be shielded from a malfunctioning or Ill forked database from core client El 17 de noviembre de 2015 16:24:42 CET, Tom Harding <tomh@thinlink•com> escribió: >On Nov 17, 2015 5:54 AM, "Tamas Blummer via bitcoin-dev" < >bitcoin-dev@lists•linuxfoundation.org> wrote: >> >> Isolating storage from the rest of consensus code is technically >desirable, but implementations using different storage will be unlikely >bug-for-bug compatible, >> hence able to split the network. > >The problem with unknown bugs is you don't know how serious they are. >A >serious bug could itself be devastating. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. [-- Attachment #2: Type: text/html, Size: 3929 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-17 22:17 ` telemaco @ 2015-11-20 14:15 ` Jorge Timón 0 siblings, 0 replies; 36+ messages in thread From: Jorge Timón @ 2015-11-20 14:15 UTC (permalink / raw) To: telemaco; +Cc: Bitcoin Dev, Gregory Maxwell On Tue, Nov 17, 2015 at 11:17 PM, telemaco via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > Shouldn't a odbc jdbc jconnect or equivalent be totally transparent for the > consensus code? Yes, but we're only testing levelDB and we couldn't assure that it won't produce unintentional consensus forks with other databases behind the whatever db-agnostic interface. I believe Bitcoin Core should officially support only one database at a time. And if that is to change in the future, I don't think it should be before a storage-agnostic libconsensus is encapsulated (and after that there will still be risks and costs in officially supporting several several databases simultaneously). As has been said, these kind of experiments are welcomed outside of bitcoin/master though. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 2:58 ` Peter R 2015-11-15 3:30 ` Gregory Maxwell @ 2015-11-16 1:52 ` Rusty Russell 1 sibling, 0 replies; 36+ messages in thread From: Rusty Russell @ 2015-11-16 1:52 UTC (permalink / raw) To: Bitcoin Dev, Bitcoin Discuss; +Cc: Gregory Maxwell Peter R via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> writes: > You are looking at the problem from a “top down” governance > perspective assuming you know what code is actually being run and what > rules the market wants. We have strayed far from both the Subject line and from making progress on bitcoin development. Please redirect to bitcoin-discuss. I have set the moderation bits on the three contributors from here down (CC'd): your next post will go to moderation. Thanks, Rusty. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 1:02 ` Peter R 2015-11-15 1:08 ` Gregory Maxwell @ 2015-11-15 3:04 ` Luke Dashjr 2015-11-15 3:17 ` Peter R 1 sibling, 1 reply; 36+ messages in thread From: Luke Dashjr @ 2015-11-15 3:04 UTC (permalink / raw) To: bitcoin-dev, Peter R; +Cc: telemaco On Sunday, November 15, 2015 1:02:33 AM Peter R via bitcoin-dev wrote: > A group of us have been exploring this “meta-cognition” idea with Bitcoin > Unlimited. For example, Bitcoin Unlimited can be (optionally) made to > automatically fork to the longest chain if it “gets stuck” and can neither > prove that a block is valid nor that the block is invalid. This situation isn't something that can be ignored and simply moved past. If you can't determine the validity of a block, you also cannot process its results correctly. Taking for example the BDB/LevelDB issue, the result was that BDB failed to accept further changes to the UTXO set. Unless the UTXO set could be updated correctly, there is no way to even attempt to validate the next block or any new transactions. Luke ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-11-15 3:04 ` Luke Dashjr @ 2015-11-15 3:17 ` Peter R 0 siblings, 0 replies; 36+ messages in thread From: Peter R @ 2015-11-15 3:17 UTC (permalink / raw) To: Luke Dashjr; +Cc: bitcoin-dev, telemaco > On Sunday, November 15, 2015 1:02:33 AM Peter R via bitcoin-dev wrote: >> A group of us have been exploring this “meta-cognition” idea with Bitcoin >> Unlimited. For example, Bitcoin Unlimited can be (optionally) made to >> automatically fork to the longest chain if it “gets stuck” and can neither >> prove that a block is valid nor that the block is invalid. > > This situation isn't something that can be ignored and simply moved past. If > you can't determine the validity of a block, you also cannot process its > results correctly. Taking for example the BDB/LevelDB issue, the result was > that BDB failed to accept further changes to the UTXO set. Unless the UTXO set > could be updated correctly, there is no way to even attempt to validate the > next block or any new transactions. Great point, Luke! Indeed, whether the program can or cannot continue after a Type 1 consensus mismatch depends on the specifics of the situation and exactly how the code was written. But I agree: there are cases where the program *can’t* continue. In those cases it would halt. This would require manual intervention to fix but avoids the problem of potential double-spends during the fork event. This would be preferable to knowingly causing a fork. Peter ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db 2015-10-29 6:57 telemaco 2015-10-29 8:03 ` Luke Dashjr @ 2015-10-29 8:17 ` Gregory Maxwell 1 sibling, 0 replies; 36+ messages in thread From: Gregory Maxwell @ 2015-10-29 8:17 UTC (permalink / raw) To: telemaco; +Cc: Bitcoin Dev On Thu, Oct 29, 2015 at 6:57 AM, telemaco via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote: > Why not "outsource" totally that data management part to the already > existing with decades of experience database world. People would be able to > create incredibly easy bitcoin statistics/graphs/analisys with existing > software packages (hey even excel or libreoffice like) or connect bitcoin > data to their own sources and if so they chose analyze bitcoin data on a > datawarehouse or any imaginable approach. Of course every transaction would > be have to do through the bitcoin node and only the data management would be > on rdbms side. The word "database" is likely confusing people here. This is not a database in an ordinary sense. The bitcoin core consensus engine requires a highly optimized ultra compact data structure to perform the lookups for coin existence. The data stored is highly compressed and very specialized, it would not be useful to other applications. Right now, on boring laptop hardware, during network synchronization updates to this database run at over 10,000 records per second, while the system is also busy doing the other validation chores of a node. This is backended by a high performance transactional key value store. The need for performance here is essential to even keeping up with the network, it's not about enabling any kind of fancy querying (bitcoin core does not offer fancy querying), it's about the base load that every node must handle to usably sync up and keep up with the Bitcoin network. The backend can be swapped out for something else that provides the same properties, but doing so does not give you any of the inspection/analytics that you're looking for. Systems that do that exist, and they require databases taking hundreds of gigabytes of storage and take days to weeks to import the network data. They're great for what they're for, but they're not suitable for consensus use in the system for space efficiency, performance, and consensus consistency reasons. ^ permalink raw reply [flat|nested] 36+ messages in thread
end of thread, other threads:[~2015-11-20 14:15 UTC | newest] Thread overview: 36+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-10-22 21:26 [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db Jeff Garzik 2015-10-22 21:54 ` Patrick Strateman 2015-10-22 21:56 ` Joseph Gleason ⑈ 2015-10-23 6:53 ` Jonas Schnelli 2015-10-23 7:45 ` Lucas Betschart 2015-10-28 20:28 ` Sean Lynch 2015-10-28 21:11 ` Jeff Garzik 2015-10-23 10:30 ` Tom Zander 2015-10-26 18:06 ` Douglas Roark 2015-10-28 15:52 ` Tom Zander 2015-11-18 0:06 ` Jonathan Wilkins 2015-10-29 6:57 telemaco 2015-10-29 8:03 ` Luke Dashjr 2015-10-30 3:04 ` Simon Liu 2015-10-30 3:35 ` Gregory Maxwell 2015-10-30 4:04 ` Peter R 2015-10-30 4:28 ` Gregory Maxwell 2015-11-15 1:02 ` Peter R 2015-11-15 1:08 ` Gregory Maxwell 2015-11-15 1:45 ` Peter R 2015-11-15 2:10 ` Gregory Maxwell 2015-11-15 2:58 ` Peter R 2015-11-15 3:30 ` Gregory Maxwell 2015-11-15 4:10 ` Peter R 2015-11-15 10:12 ` Jorge Timón 2015-11-15 11:28 ` Jorge Timón 2015-11-15 15:48 ` Peter R 2015-11-15 17:06 ` Peter R 2015-11-17 13:54 ` Tamas Blummer 2015-11-17 15:24 ` Tom Harding 2015-11-17 22:17 ` telemaco 2015-11-20 14:15 ` Jorge Timón 2015-11-16 1:52 ` Rusty Russell 2015-11-15 3:04 ` Luke Dashjr 2015-11-15 3:17 ` Peter R 2015-10-29 8:17 ` Gregory Maxwell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox