--- Day changed Thu Oct 22 2015
00:38 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 265 seconds]
00:40 -!- ParadoxSpiral_ [~ParadoxSp@p508B8D51.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
00:43 -!- ParadoxSpiral [~ParadoxSp@p508B9A93.dip0.t-ipconnect.de] has quit [Ping timeout: 260 seconds]
01:11 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
01:13 < phantomcircuit> can confirm bitcoin core requires reindex on power failure under windows
01:14 < phantomcircuit> probably leveldb env driver is stupid
01:15 < gmaxwell> phantomcircuit: fixy fixy?  unfortunately I think it's unmaintained.
01:15 < gmaxwell> as chrome uses some chrome specific abstractions for IO. :(
01:19 < phantomcircuit> gmaxwell, i can take a look at it, probably it should just be replaced with the current posix version which is just fopen/fread/fwrite basically
01:24 < GitHub16> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/0fbfc5106cd9...a09297010e17
01:24 < GitHub16> bitcoin/master 579b863 Wladimir J. van der Laan: devtools: Add security-check.py...
01:24 < GitHub16> bitcoin/master a092970 Wladimir J. van der Laan: Merge pull request #6854...
01:24 < GitHub129> [bitcoin] laanwj closed pull request #6854: devtools: Add security-check.py (master...2015_10_security_checks) https://github.com/bitcoin/bitcoin/pull/6854
01:25 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 240 seconds]
01:26 < wumpus> phantomcircuit: yeah https://github.com/bitcoin/bitcoin/issues/5610
01:26 < wumpus> no clue on solving it though
01:26 < wumpus> for some reason most people with a clue about windows internals seem to be blackhats, and they like things broken :p
01:27 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
01:27 < phantomcircuit> wumpus, it's using windows memory mapping
01:27 < gmaxwell> phantomcircuit: reports went up astronomically after 0.10 and some users reported it worked fine for them before them and fails every time since.
01:27 < phantomcircuit> im going to assume it's that
01:28 < gmaxwell> phantomcircuit: oh it's writing via mmap too?
01:28 < phantomcircuit> i just checked and there's a Win32Map something or other 
01:28 < phantomcircuit> i assume it's using that
01:29 < gmaxwell> https://litteration.files.wordpress.com/2013/05/science-dog.jpg
01:29 < wumpus> it has been a while since we updated leveldb, maybe some things got fixed since?
01:30 < wumpus> heh
01:30 < phantomcircuit> gmaxwell, that's about right
01:31 < phantomcircuit> it's windows... how do you even begin to debug this
01:32 < wumpus> that's a dark art
01:33 < wumpus> wouldn't be surprised if it requieres at least some, possibly expensive, proprietary software applications
01:35 < wumpus> debugging-by-mutating-the-code may be the most promising approach, extract a bare bones program that uses leveldb and crashes, then change it (adding flushes everywhere) until it manages to not corrupt anymore
01:36 < gmaxwell> it's 100% reproducable it seems, which helps.
01:36 < wumpus> this is the approach that works for fixing problems with underdocumented hardware, it should work for windows too...
01:36 < gmaxwell> the fact that it went way up when we turned up checking suggests to me that it's something like a truncated record at the end that was otherwise silently recoverable.
01:37 < gmaxwell> e.g. something like the osx bug.
01:38 < wumpus> yes good point - can you send me a database that is corrupted by this problem?
01:39 < gmaxwell> phantomcircuit: you would be you, as I don't have a repro.
01:40 < phantomcircuit> heh
01:41 < phantomcircuit> gmaxwell, i suspect you're right that it's similar to the OSX bug
01:41 < phantomcircuit> i also suspect it would be easier to replace leveldb with lmdb
01:41 < wumpus> phantomcircuit:  can you send me a database that is corrupted by this problem?
01:41 < gmaxwell> mmap only, goodbye 32 bit hosts.
01:41 < gmaxwell> Among other limitations. :(
01:41 < Arnavion> It's funny because if you ask Windows devs they'll tell you the dev tooling on Windows is far better than on Linux
01:42 < phantomcircuit> wumpus, i mindlessly clicked the reindex button
01:42 < gmaxwell> Arnavion: there is some aspect of "tools you know are always better than ones you don't"
01:42 < phantomcircuit> i can certainly try to do so though
01:42 < wumpus> Arnavion: it may be more user friendly on windows, but it's harder to get insight about what is happening on a system
01:42 < Arnavion> There's also the aspect of GUI tools that don't suck
01:42 < Arnavion> but I'm biased
01:42 < gmaxwell> Arnavion: you mean GUI tools inherently sucking? :P
01:43 < Arnavion> Oh, but that's what WinDbg is for
01:43 < Arnavion> You can do everything with a keyboard if you wish
01:43 < wumpus> anyhow this is not constructive
01:44 < wumpus> if you feel helpful, please demonstrate your skills by solving the leveldb corruption issue Arnavion :)
01:44 < Arnavion> I have no skills
01:45 < Arnavion> but I can take a look
01:45 < wumpus> demonstrate the great capability of windows debugging tools, then
01:46 < wumpus> phantomcircuit: but you can reproduce it! just start a new datadir, crash the thing, and send it to me
01:51 < wumpus> lmdb is not the panacea some people think - but if you feel like patching bitcoind to use it, that'd be pretty neat as we could run some benchmarks and comparisons
01:59 < phantomcircuit> wumpus, the main thing that's missing from leveldb is... sequence numbers in the journal
01:59 < phantomcircuit> if you miss a full record of writes it doesn't know
02:00 < wumpus> still, we need to debug this issue, as it is clearly windows specific, so not fundamental to leveldb
02:00 < wumpus> if this was broken on all platforms I'd agree that looking for a replacement would be advisable...
02:01 < phantomcircuit> wumpus, it *is* broken on all platforms
02:01 < phantomcircuit> the failure rate is simply much lower
02:02 < gmaxwell> one step at a time.
02:02 < gmaxwell> lets making the bleeding stop on windows first.
02:03 < wumpus> that's a nihilistic way of looking at it phantomcircuit, I like that, yes, it's all broken, some things with a very low failure rate... but it doesn't help solving the issue :)
02:04 -!- rubensayshi [~ruben@91.206.81.13] has joined #bitcoin-core-dev
02:05 < wumpus> I'm sure lmdb is also broken in some subtle ways
02:06 < wumpus> anyhow, if you're not going to send me a corrupted datbase I'm going to start working on something else
02:07 < gmaxwell> I assume he has to wait for the reindex to complete to reproduce. :P
02:08 < wumpus> wha?
02:08 < wumpus> I assumed it would also happen when crashing during reindex?
02:08 < Luke-Jr> wumpus: what is Windows specfic?
02:09 < Luke-Jr> [08:13:52] <phantomcircuit> can confirm bitcoin core requires reindex on power failure under windows <-- I can confirm it requires reindex on power failure under Linux..
02:09 < wumpus> in that case, sorry, no hurry implied
02:10 < Luke-Jr> wumpus: if you in fact want such a corrupt bitcoin dir, I can probably get one for oyu
02:10  * Luke-Jr ponders where he put his USB Armory
02:11 < Luke-Jr> oh, it's plugged in
02:11 < wumpus> Luke-Jr: great!
02:12 < phantomcircuit> Luke-Jr, ive never had that issue on linux
02:12 < gmaxwell> it's interesting that the usb armory corrupts, perhaps not the same issue as windows though (though perhaps also worth fixing) 
02:14 < wumpus> indeed, may be a different issue
02:22 < Luke-Jr> FWIW, removing power immediately after IBD started did NOT reproduce it
02:22 < Luke-Jr> anything in debug.log to look for to know when it does the first flush to disk?
02:23 < wumpus> set a low dbcache to force lots of flushes?
02:23 < wumpus> AFAIK it doesn't flush unless necessary until the initial block download is complete
02:25 -!- BashCo [BashCo@gateway/vpn/mullvad/x-jrahxfhdwhsxxpve] has joined #bitcoin-core-dev
02:27 < Luke-Jr> 'often' is too slow
02:28 < Luke-Jr> 2015-10-22 09:27:59 UpdateTip: new best=000000003f7e074587fa1684ac863519fea3c64040b05ddd04948a13f7b19b42  height=415  log2_work=40.700462  tx=423  date=2009-01-14 05:56:05 progress=0.000002  cache=419
02:33 < Arnavion> From just lazy-mode browsing the code, Win32MapFile::Sync() flushes but Win32MapFile::Flush() is a no-op, whereas the equivalent functions for posix both do things
02:34 < Arnavion> I did not see how they are called from common code to see if that matters
02:37 < wumpus> it all depends on whether this map file is used for writing
02:38 < wumpus> AFAIK on other OSes, writing uses normal file system commands, whereas reading can use mmap
02:38 < wumpus> if t his is different on windows that would explain something
02:39 < Arnavion> Windows has a dedicated FlushViewOfFile function to flush a mapped region
02:39 < Arnavion> That is called by Sync(), but not by Flush()
02:58 < Luke-Jr> hmm
02:58 < Luke-Jr> it seems I can't reproduce during IBD or something
03:07 < Luke-Jr> and a fully synced state is huge :/
03:07 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Quit: Leaving]
03:08 < midnightmagic> Luke-Jr: you're reproducing on a Windows OS?
03:08 < Luke-Jr> midnightmagic: no
03:18 -!- jgarzik [~jgarzik@104-178-201-106.lightspeed.tukrga.sbcglobal.net] has joined #bitcoin-core-dev
03:18 -!- jgarzik [~jgarzik@104-178-201-106.lightspeed.tukrga.sbcglobal.net] has quit [Changing host]
03:18 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev
03:25  * Luke-Jr ponders
03:26 < Luke-Jr> wumpus: well, the last chainstate I had before today is 922 MB and demanded a reindex.. not sure if that's good enough, because I can't upload 44 GB :/
03:27 < wumpus> that's kind of huge - do you perhaps know what ldb file the corruption is in?
03:27 < Luke-Jr> no, I have no idea how leveldb works :/
03:28 < Luke-Jr> I could probably just give you access to the device?
03:39 < Luke-Jr> wumpus: see PM for login info
03:39 < wumpus> ok
03:40 < Luke-Jr> setup auth for your github ssh key; let me know if there's a better one
03:40 < Luke-Jr> wumpus: the leveldb in question is under ~/.bitcoin.bak
03:41 < Luke-Jr> perhaps relevant: tar: /home/usbarmory/.bitcoin.bak/chainstate/1463292.ldb: File shrank by 2020259 bytes; padding with zeros
03:42 < wumpus> uh...
03:42 < wumpus> yes, that doesn't sound good. Were you tarring while bitcoin running?
03:43 < Luke-Jr> not sure if bitcoind is running, but if it is, it's not running on *this* database
03:43  * Luke-Jr wonders what it's doing so much to hang SSH
03:44 < wumpus> ok, it would not be strange if it is happening while running tar on the datadir that bitcoin was running in, those files are deleted and recreated all the time
03:44 < Luke-Jr> right
03:45 < sipa> wumpus: leveldb seems unmaintained
03:47 < wumpus> bleh
03:52 < wumpus> I'm happy we at least never decided to use it for the wallet
03:53 < sipa> maybe we should switch to sqlite... at least that's very well tested and maintained
03:54 < wumpus> it is, but it's not very fast for key/value storage, and I don't think it handles such large databases very well
03:55 < btcdrak> sipa: sqlite is very slow
03:56 < wumpus> if you need more advanced query features it is very nice though
03:57 < btcdrak> it would be so much easier and flexible if we were using SQL of some kind...
03:57 < wumpus> what would be easier?
03:57 < sipa> btcdrak: leveldb's own benchmark says that sqlite is faster for random reads
03:57 < btcdrak> sipa: interesting
03:57 < wumpus> for wallet metadata sql would be reasonably useful
03:58 < sipa> writes are much slower, but we do very few writes
03:58 < btcdrak> wumpus: well you get a lot of stuff for free with a SQL database stack.
03:58 < sipa> such as?
03:58 < Luke-Jr> # dmesg
03:58 < Luke-Jr> Segmentation fault
03:58 < Luke-Jr> -.-
03:59 < wumpus> during initial sync we do quite a lot of writes (every coin that is touched is written back), after that, most are reads
03:59 < btcdrak> sipa: indexes for a start. the ability to query the dataset externally from bitcoind etc
03:59 < wumpus> using database indexes would mean using a very verbose format
03:59 < sipa> btcdrak: we're not storing any data in a way that's useful to index
04:00 < sipa> we'd lose massive performance by expanding the data
04:00 < wumpus> that kind of defeats the purpose - bitcoind's database should optimized for its own purposes as running a node, not external ones
04:00 < btcdrak> I had wondered about leveldb's ongoing support. it really looks like an orphaned project at this time.
04:01 < wumpus> sipa: right
04:01 < jgarzik> btcdrak, the expanded dataset in SQL is something like 40GB at least
04:01 < jgarzik> not including the block data itself
04:02 < jgarzik> sipa, worse comes to worse, we can do our own key/value store...
04:02 < wumpus> also bitcoin's databases are not an external interface, don't query them directly unless it's to write troubleshooting/recovery tools
04:03 < jgarzik> there are actually a few optimizations you can make if you assume your keys are hashes already
04:03 < jgarzik> not that seeks matter these days, but some seeks can be eliminated
04:04 < wumpus> reinventing the wheel should be  the last recourse
04:04 < Luke-Jr> wumpus: whatever just happened appears to have fried the device, so I guess it'll be a while :/
04:04 < jgarzik> <grin> https://github.com/jgarzik/pgdb https://github.com/jgarzik/pagedb
04:04 < wumpus> Luke-Jr: I hope it wasn't me logging in! :p
04:05 < Luke-Jr> heh, I don't see how that could do it
04:05 < sipa> leveldb does have a benchmark about batch write performance
04:05 < sipa> where sqlite is much slower
04:05 < wumpus> I don't see either, I didn't even get a prompt
04:05 < wumpus> I've disconnected now
04:05 < sipa> though sqlite does win in synchronous write speed, which we also do frequently
04:06 < Luke-Jr> wumpus: makes me wonder if it's just a bad microSD card, which could make the failures invalid
04:06 < wumpus> would be interesting to patch sqlite into bitcoind and compare and do benchmarks
04:06 < sipa> the only reason i'm suggesting sqlite is because it is *very* well tested afaik
04:07  * wumpus says that for the second time today about a different database
04:07 < wumpus> sipa: also on windows? 
04:07 < wumpus> leveldb is also very well tested, it is used by many companies in production... just mostly on linux/unix servers
04:08 < sipa> yes
04:10 < wumpus> do you know, how does sqlite do at caching?
04:10 < btcdrak> sipa: wumpus: maintenance is probably more of a concern than anything else. I do remember wondering about leveldb's longevity. sqlite is way too popular to disappear.
04:10 < jgarzik> sqlite leans a lot of OS caching, temporary files etc.
04:10 < wumpus> jgarzik: it can't do a much worse job than leveldb at that at least
04:11 < btcdrak> if we're just doing key/value what about all those database that do that sort of thing, like those nosql stacks?
04:12 -!- PRab [~chatzilla@2601:40a:8000:8f9b:bd13:f19b:c988:ff92] has joined #bitcoin-core-dev
04:12 < jgarzik> I'm a big fan of sqlite, and used it in my DNS server project.   I don't know that sqlite has batch update - does begin/commit suffice given our workflow?  They are -mostly- equivalent but not 100% equiv
04:12 < btcdrak> still. sqlite would be a better choice. But here's the thing, if we use sqlite then you're opening up the door for just about any database backend. specially if we made an abstraction layer
04:12 < wumpus> "The maximum size of a database file is 2147483646 pages. At the maximum page size of 65536 bytes, this translates into a maximum database size of approximately 1.4e+14 bytes "   ok, apparently I misremembered
04:13 < wumpus> btcdrak: there aren't that much embeddable key/value stores 
04:13 < btcdrak> wumpus: true.
04:13 < btcdrak> and sqlite wins in terms of support/maintenance.
04:13 < jgarzik> I certainly wouldn't suggest expanding the bitcoind db in SQL, but using it as a dumb blob datastore shouldn't be a big issue
04:13 < wumpus> and most of them are likely one-off projects, even worsely maintained than leveldb
04:13 < wumpus> right
04:13 < wumpus> jgarzik: +1
04:13 -!- PRab_ [~chatzilla@2601:40a:8000:8f9b:6059:f864:4ad7:aa7b] has quit [Ping timeout: 240 seconds]
04:13 < btcdrak> +!
04:14 < btcdrak> +1
04:14 < Luke-Jr> reminder: UTXO db is consensus-critical.
04:15 < sipa> very aware
04:15 < wumpus> sqlite is extremely easy to embed in a project, it's just one C file
04:15 < jgarzik> kv datastores off the top of my head: leveldb, gdbm, [n]dbm, tokyo cabinet, kyoto cabinet, berkeley db
04:15 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
04:15 < jgarzik> wumpus, er huh?
04:15 < Luke-Jr> wumpus: surely one *big* C file. all of SQL can't be trivial.
04:15 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
04:15 < jgarzik> wumpus, sqlite is quite bigger than one C file
04:16 < wumpus> hm it used to be at least
04:16 < jgarzik> berkeley db is well maintained :) :)
04:17 < Luke-Jr> sqlite is like 140kLOC
04:18 < jgarzik> for sqlite, if you pre-compile the SQL queries, it goes pretty fast
04:18 < sipa> of course we'd precompile them...
04:19 < sipa> i din't think git subtreeing sqlite is reasonable
04:20 < wumpus> oh this is why I thought it was one C file, yes it's a very large one with everything pasted together: https://www.sqlite.org/amalgamation.html
04:20 < wumpus> sipa: license-wise?
04:20 < sipa> size wise
04:21 < jgarzik> sqlite is public domain, one of the few
04:22 < sipa> oh, that amalgation is pretty awesome
04:22 < sipa> we can compile it with various flags for extensions disabled
04:22 < wumpus> apparantly leveldb is 26kLOC in total
04:22 < wumpus> sipa: yep
04:24 < jgarzik> imo LOC and size are secondary factors
04:24 < jgarzik> primary is on going maintenance, reliability, ...
04:24 < wumpus> LOC would be important if there is the chance we end up having to maintain/troubleshoot it ourself
04:24 < wumpus> (like now for leveldb)
04:25 < jgarzik> indeed
04:25 < jgarzik> I think pgdb will likely be 10k loc when finished
04:27 < wumpus> "SQlite as a  key-value database" https://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase
04:28 < wumpus> looks very easy to do, the only question is indeed jgarzik's whether the TRANSACTION/COMMIT is equivalent to batching as we use it now
04:29 < jgarzik> another sql caveat - based on experience - you have to be careful that your 'strings' are not translated in any way by the engine via unicode etc.
04:29 < Luke-Jr> jgarzik: nobody is going to maintain consensus-critical behaviour
04:29 < wumpus> sql injections? *ducks*
04:29 < jgarzik> I used sqlite to store binary DNS records
04:29 < sipa> sqlite is fully transactional
04:29 < Luke-Jr> no matter what we use, we will need to maintain it
04:30 < Luke-Jr> even if that just means reviewing each upstream release
04:30 < sipa> or just reviewing their test practices
04:30 < jgarzik> sqlite is very well hammered
04:30 < wumpus> (no, sqlite has proper parametrized queries, and from my experience its BLOBs are binary clean)
04:30 < jgarzik> few maintain software to consensus critical standards though
04:31 < sipa> agree, but a database interface clearly should... it's perfectly specified in both dirextion: it should find every record that exist, and not find any reorc that doesn't exist
04:31 < jgarzik> another bit - make sure there are no limitations on rows-modified-at-once
04:31 < sipa> so anything that is nkt consensus compatible is a bug
04:31 < jgarzik> otherwise we re-introduce the BDB fork issue (locks)
04:32 < Luke-Jr> sipa: fixing bugs is a consensus compatibility bug!
04:32 < sipa> jgarzik: the "bug" in the bdb case was that we didn't chefk tue bdb return value, and regarded failure to write as a block validation failure
04:33 < sipa> Luke-Jr: fully agree there
04:33 < Luke-Jr> anyhow, I need to go to sleep, but I think we will regret it if we import sqlite to consensus-critical code.
04:33 < jgarzik> sipa, well there was an issue where we were hitting BDB max lock
04:33 < jgarzik> sipa, the point still stands - sql engines do somethings have max-rows-updated-in-one-transaction type limits
04:33 < sipa> jgarzik: sure, but that wouldn't have been a consensus failure threat if we didn't treat out-of-locks as invalid-block
04:34 < jgarzik> *sometimes
04:34 < sipa> jgarzik: it was us who turned an administrative restriction into a consensus problem
04:34 < jgarzik> so, sql transaction limits is another one for the eval list
04:35 < sipa> if we would just have gone "oops! can't write! quitting", like we do for out of disk, bdb would not have caused forks
04:35 < wumpus> right, if it would treat database errors as a fatal error instead of rejection, it'd wouldn't have been as bad
04:35 < jgarzik> nod
04:35 < sipa> it would have been a bad DoS attack, though
04:35 < jgarzik> yep
04:35 < sipa> but not a fork
04:35 < wumpus> but fairly easy to resolve
04:35 < wumpus> right
04:36 < sipa> of course, you are absolutely right we need to be aware of such limits in sqlite or whatever we're incestigating
04:36 < sipa> eh, investigating
04:36 < jgarzik> lol
04:36 < sipa> the keys are like right nezt to each other
04:37 < sipa> also, sqlite databases are single files, right?
04:38 < jgarzik> yes*
04:38 < jgarzik> * some temporary files such as journals are created along the way, and must exist for recovery post-crash
04:38 < sipa> ok, similar to bdb
04:39 < sipa> that's fine for application database stuff
04:39 < sipa> maybe not wantes for wallets
04:39 < sipa> *wanted
04:40 < jgarzik> sqlite files do want periodic maintenance via 'VACUUM'
04:41 < sipa> we can do that in our flushtodisk function
04:41 < jgarzik> they get fragmented, old records cluttered, performance degrades a bit over time
04:41 < jgarzik> it is a very, very heavyweight operation
04:42 < jgarzik> info: https://sqlite.org/lang_vacuum.html
04:42 < jgarzik> auto_vacuum handles large deletes, but not fragments over time
04:43 < jgarzik> could just run it at startup and then auto_vacuum
04:43 < sipa> ok
04:56 < btcdrak> I've not seen everyone get so excited about something for a long time.
04:58 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.]
05:01 < wumpus> "The author disclaims copyright to this source code.  In place ofa legal notice, here is a blessing:   May you do good and not evil.  May you find forgiveness for yourself and forgive others.  May you share freely, never taking more than you give." I love sqlite's appraoch to licensing
05:11 < btcdrak> wumpus: if only there were more like that
06:06 < jgarzik> At https://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase sqlite seems across the board worse... except for large numbers of records (our use case)
06:06 < jgarzik> I'm happy to create an sqlite branch for bitcoind, if nobody else is working on it
06:07 -!- pigeons [~pigeons@94.242.209.214] has quit [Ping timeout: 240 seconds]
06:09 -!- pigeons [~pigeons@94.242.209.214] has joined #bitcoin-core-dev
06:09 -!- pigeons is now known as Guest89666
06:10 < btcdrak> jgarzik: go for it
06:13 < sipa> how mature is sqlite4?
06:14 < jgarzik> good question
06:14 < jgarzik> vast majority in field is 3
06:14 < sipa> its design seems much closer to leveldb
06:14 < sipa> as it is... a key/value store internally
06:15 < sipa> i wonder if there is an api to access that key/value store interface
06:15 < sipa> i have read before that you can
06:15 < wumpus> I was starting on it, but will happily leave it to you jgarzik
06:15 < sipa> The default built-in storage engine is a log-structured merge database. It is very fast, faster than LevelDB, supports nested transactions, and stores all content in a single disk file.
06:16 < sipa> ^ from the sqlite4 design
06:16 < wumpus> where do you seea nything about sqlite4? at least the download page only has 3
06:16 < sipa> ok, sqlite4 is far from released
06:17 < wumpus> oh, "trunk" :-)
06:17 < sipa> not an option at this point
06:17 < wumpus> sounds great though
06:17 < sipa> alsoz sqlite3 does not promise backward write compatibility... so it is like bdb, and we need to guarantee only increases in version
06:18 < wumpus> it does promise backward read compatibility?
06:18 < sipa> yes
06:19 < wumpus> well then it's not too bad, if it detects a downgrade, it could export|import the database
06:19 < sipa> wait
06:19 < wumpus> berkeleydb doesn't even guarantee that much
06:19 < sipa> databases created by 3.3 can't be read by earlier 3.x versions
06:20 < sipa> lol, and this caused so much problems they reverted to the old format by default in 3.3.6
06:21 < jgarzik> note - single file db implies large file support req
06:22 < sipa> is that not available on every platform these days?
06:22 < wumpus> is that unreasonable these days?
06:22 < jgarzik> every platform, but some filesystems may be e.g. fat32 for weird reasons
06:22 < wumpus> ... and has been for 10 years or so
06:23 < jgarzik> I think it's reasonable
06:23 < sipa> oh: a database created by version X will always be read/write compatible with version X, even if modified by later versions
06:23 < jgarzik> just noting
06:23 < wumpus> fat32 is pretty much dead
06:23 < sipa> so it's not an auto-update to whatever the format in the later version is, like bdb
06:23 < jgarzik> sipa, yeah, kernel filesystem rules
06:23 < wumpus> it also doesn't support bigger disks
06:23 < jgarzik> wumpus, sadly very much alive on USB sticks
06:24 < sipa> put bitcoind'd chainstate on a USB stick: you are eaten by a grue
06:24 < wumpus> *small* USB sticks
06:24 < sipa> also: for now the chainstate still fits in 2 GB...
06:24 < sipa> ... for now
06:25 < jgarzik> as I said... just making a note because it's something of which people should be aware.  continuing to hack on it :)
06:25 < sipa> awesome, thanks
06:25 < wumpus> great
06:27 < wumpus> 32GB is pretty much the limit for FAT32 devices, it's possible to have larger volumes, but it becomes really ineffiecient and at least windows doesn't allow formatting them
07:06 -!- aj [aj@cerulean.erisian.com.au] has quit [Remote host closed the connection]
07:06 -!- lightningbot [supybot@2400:8900::f03c:91ff:fedf:3a06] has quit [Remote host closed the connection]
07:10 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
07:26 -!- fanquake [~Adium@unaffiliated/fanquake] has joined #bitcoin-core-dev
07:33 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has joined #bitcoin-core-dev
07:33 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has quit [Changing host]
07:33 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev
07:40 -!- malte [~malte@alkaid.uberspace.de] has quit [Ping timeout: 250 seconds]
07:40 -!- fanquake [~Adium@unaffiliated/fanquake] has quit [Quit: Leaving.]
07:41 -!- lightningbot [supybot@2400:8900::f03c:91ff:fedf:3a06] has joined #bitcoin-core-dev
07:44 -!- Greg__ [~greg@h105.31.40.69.dynamic.ip.windstream.net] has joined #bitcoin-core-dev
07:53 -!- malte [~malte@2a00:d0c0:200:0:b9:1a:9c2c:1] has joined #bitcoin-core-dev
07:54 < GitHub122> [bitcoin] MarcoFalke closed pull request #6866: [trivial] fix white space in rpc help messages (master...MarcoFalke-2015-rpcWhitespace) https://github.com/bitcoin/bitcoin/pull/6866
07:54 < morcos> sipa: i'm trying to think about how to preserve the state of the mempool while generating block templates
--- Log closed Thu Oct 22 08:00:37 2015
--- Log opened Thu Oct 22 08:01:11 2015
08:01 -!- kanzure_ [~kanzure@unaffiliated/kanzure] has joined #bitcoin-core-dev
08:01 -!- Irssi: #bitcoin-core-dev: Total of 75 nicks [0 ops, 0 halfops, 0 voices, 75 normal]
08:01 -!- Eliel_ [~jojkaart@jkaartinen.iki.fi] has joined #bitcoin-core-dev
08:02 < sipa> morcos: how about being able to mark mempool txn as locked?
08:03 < sipa> individual transactions, that is
08:03 < morcos> sipa: how do you mean?  the block template generation code needs to mark all of them as locked?
08:03 < morcos> or do you mean if you need to delete something, such as with RBF, then you mark it as deleted and go ahead and add the new one?
08:04 -!- __Greg [~greg@h105.31.40.69.dynamic.ip.windstream.net] has joined #bitcoin-core-dev
08:04 < sipa> morcos: while the template generator runs, it locks transactions one by one as they are added to the template
08:04 -!- teward- [teward@ubuntu/member/teward] has joined #bitcoin-core-dev
08:04 -!- banana_lotus [~BananaLot@54.186.186.141] has joined #bitcoin-core-dev
08:04 < sipa> so the next transactions it fetches are guaranteed to be not conflicting with the ones it already has
08:05 < sipa> you want a new incoming block to override that though, and cancel the generator to make it start over
08:05 < morcos> yes agreed with that last part, i'll tackle that later
08:06 -!- guruvan- [~guruvan@unaffiliated/guruvan] has joined #bitcoin-core-dev
08:06 < morcos> but i'm not sure how to just lock txs one by one, you'd have to lock the mempool.cs each time you went to look up a tx to decide to add it to the mempool
08:06 < morcos> i'm starting off by trying to modify the existing CNB to hold locks a lot less
08:06 -!- Netsplit *.net <-> *.split quits: cfields_, Eliel, teward, BananaLotus, Greg__, lightningbot, murr4y, PRab, molly, kanzure,  (+2 more, use /NETSPLIT to show all of them)
08:06 -!- teward- is now known as teward
08:06 -!- banana_lotus is now known as BananaLotus
08:06 -!- guruvan- is now known as guruvan
08:07 < morcos> so i'm imagning it basically copies ptrs to all the txs in the mempool and the score it needs for them
08:07 < morcos> then it doesn't need to hold mempool.cs any more as long as the ptr does not become invalid
08:07 < morcos> which would only happne if the txn is deleted
08:08 < sipa> i guess transactions would get a refcount
08:08 < sipa> and the refcount value itself would be protected by mempool.cs
08:08 < morcos> but i think that has to be for all of the txs as you're running the logic of figuring out which ones need to be in the template
08:08 < morcos> yeah, but i was imagining it was a singular refcount
08:08 < sipa> hmm, does the mempool not hold some index sorted by template inclusion preference?
08:08 < morcos> instead of per tx
08:09 -!- Irssi: Join to #bitcoin-core-dev was synced in 533 secs
08:09 < morcos> yeah so thats where my approach might not be the best approach b/c the new template generation code will have such an index
08:09 < sipa> another idea is to remove the storage if txn from the mempool entirely
08:10 < morcos> but even still you need to iterate far past the size of a block 
08:10 < sipa> you just get a transaction store manager, which you give transactions, and it returns a refcounted pointer
08:10 < sipa> the mempool stores the pointers
08:10 < sipa> i guess that's just smart pointers
08:11 < sipa> so the template generator would grab pointers to the top N (some multiple of a block), increase their refcount, release memlool.cs
08:11 < sipa> then goes off to build a block, verify if wanted, ...
08:11 < sipa> and then release the pointers
08:12 < sipa> the mempool can change during that time... you just know that the set you grabbed at that point while holding the lock is internally consistent
08:12 < sipa> and consistent with the block is was building on
08:12 < morcos> yes thats what i'm going for
08:13 < morcos> i guess it depends on how big the set is compared to the whole mempool
08:13 < morcos> so my idea was that rather than marking individual txs
08:13 -!- Netsplit over, joins: PRab
08:13 < morcos> you just tell the mempool, hey, referring to things, don't delete
08:13 < morcos> and then occasionally that gets freed, and anything marked for deletion can happen
08:13 < sipa> hmm
08:13 < sipa> not sure
08:13 < morcos> since deletes are rarer and smaller than adds
08:13 < sipa> sounds more complicated
08:14 < morcos> ok, maybe i'll explore both
08:15 < morcos> next question and this might be related to CSV
08:15 < morcos> really annoying that CheckInputs needs cs_main
08:15 < morcos> doesn't seem like it should have to?
08:16 < sipa> so i think it's relative easy this way: turn the storage of txn in the mempool into smart pointers, and you can very cheaply and efficiently ask the mempool for the set of all its transactions
08:16 < morcos> you're storing the hashblock that its valid at, and that can't change height
08:16 < sipa> it shouldn't be needed, indeed
08:17 < morcos> sipa: not that easy, you need stuff from the mempool entries, thats where fees are stored
08:17 < morcos> so you still have to iterate to figure out which pointers you need and copy the meta information
08:17 < sipa> ok, so that fee information needs to be inside the smart pointed-to objects
08:18 < morcos> so all you've done now is taken the multi-index approach and say ehh forget that, lets just have multiple indexes referring to this same set of pointers
08:19 < sipa> i may be missing something
08:19 < morcos> i think the only complication from my idea is damn RBF... otherwise you just don't call any eviction code except periodically...  and there is no problem running it only periodically
08:19 < morcos> and you don't even have to track anything
08:20 -!- CodeShark__ [CodeShark@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
08:20 < sipa> what i'm suggesting is a way to make cheap snapshots of subsets of the mempool at a given time
08:20 < sipa> those snapshots themselves don't have an index, but can be ordered by one at the time they are created
08:20 < sipa> isn't that enough?
08:21 < morcos> yeah maybe, i think i'm just leary of the double indirection now where the multi-index is a multi-index of ptrs, but maybe thats something stupid to worry about
08:25 < sipa> the transactions inside the mempool already have a dozen or so indirections
08:26 < sipa> ok, more like 6
08:26 < sipa> but still!
08:27 < morcos> so something like a boost::shared_ptr
08:28 < sipa> yes, that specifically :)
08:28 < sipa> sorry, i thought it was called smart_ptr :)
08:29 < sipa> you probably want to wrap CTransaction inside something that also contains its (direct) fee and perhaps other immutable statistics
08:29 < sipa> and then have sharee_ptrs to those inside the mempool entries
08:29 < sipa> heh, we could even serialize the transactions inside to reduce memory usage...
08:31 < morcos> yes, so then it'll get a bit more complicated with ancestor package tracking when you have mutable state like ancestor fee rate that you want to copy to your template generation code, but that was going to be difficult period
08:32 < sipa> why do you need it in the template generator code?
08:32 < morcos> thats the sort the logic uses
08:33 < morcos> yikes
08:33 < sipa> sort the list of copied pointers according to the sorting criterion you want while creating it (so while holding mempool.cs)
08:33 < sipa> then release mempool.cs and you can forget it
08:33 < morcos> ok, but see thats the complicated part that takes a long time
08:33 < sipa> ah, ok
08:34 < morcos> the tx thats sorted at the top , really links to a lot of other txs , whose ptrs you have to grab to
08:34 < sipa> hmm, right
08:34 < morcos> but then how do you solve the problem of once you include that in your template
08:34 < morcos> everything elses sort changes
08:34 < sipa> i wouldn't bother with changing sorts
08:36 < morcos> well that'll be the magic of the algorithm, it'll probably have to be some heuristic, but its very easy to have expensive chain A-B-C and so C is sorted first, but then cheap C2 is still sorted quite high.
08:36 < morcos> uh not exactly but close
08:36 < morcos> but yes, i agree we don't have to be perfect
08:37 < morcos> ok you've given me some ideas... let me see what i come up with
08:40 < sipa> great
08:40 < sipa> feel free to completely disregard my ideas :)
08:48 < jgarzik> RE locking transactions one-by-one - that is the typical parallelism solution
08:48 < jgarzik> just need a gatekeeper (core struct lock) for insert/delete/move
08:53 < GitHub181> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/a09297010e17...2cd020d054d6
08:53 < GitHub181> bitcoin/master 3cb56f3 Daniel Cousens: *: alias -h for --help
08:53 < GitHub181> bitcoin/master 2cd020d Wladimir J. van der Laan: Merge pull request #6846...
08:53 < GitHub119> [bitcoin] laanwj closed pull request #6846: [Trivial] bitcoind: alias -h for -help (master...aliash) https://github.com/bitcoin/bitcoin/pull/6846
09:02 < GitHub145> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2cd020d054d6...f2c869aef2e7
09:02 < GitHub145> bitcoin/master c6824f8 J Ross Nicoll: Add DERSIG transaction test cases...
09:02 < GitHub145> bitcoin/master f2c869a Wladimir J. van der Laan: Merge pull request #6848...
09:02 < GitHub77> [bitcoin] laanwj closed pull request #6848: Add DERSIG transaction test cases (master...bip66-tests) https://github.com/bitcoin/bitcoin/pull/6848
09:15 -!- BashCo [BashCo@gateway/vpn/mullvad/x-jrahxfhdwhsxxpve] has quit [Remote host closed the connection]
09:18 < gavinandresen> FYI: git HEAD master isn't working for me on OSX.  Running make check gets me:
09:18 < gavinandresen> libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >: boost: mutex lock failed in pthread_mutex_lock: Invalid argument
09:20 < gavinandresen> I'm very confused, because git bisect blames a commit that works....
09:20 < cfields> gavinandresen: sounds like your boost may be built against the wrong stdlib? checking master on osx here now.
09:21 < gavinandresen> cfields: I thought that too, did a brew uninstall / brew install of boost
09:25 < btcdrak> morcos: sorry was afk. My other question was how does mempool evictions play into the mix? I assume one doesnt want to evict a txn that you're trying to mine.
09:26 < morcos> btcdrak: yes, thats what my concern was.  the only way a pointer to a tx becomes invalid now is if the tx is deleted.  A) throught being mined, in which case you want to abort block template generation anyway or B) through eviction or RBF
09:27 < morcos> for B1) eviction, i thought it would just be easy to just skip eviction if the template_tx_lock is held, and just make sure eviction gets to run every so often.
09:28 < morcos> but B2) RBF, doesn't have an ideal solution,  i wanted to just flag those txs, but maybe sipas idea with shared_ptrs is cleaner
09:34 < cfields> gavinandresen: ok here. What compiler/boost/osx version?
09:34 -!- BashCo [BashCo@gateway/vpn/mullvad/x-himpyklvkjsseysc] has joined #bitcoin-core-dev
09:45 < gavinandresen> cfields: Apple LLVM version 7.0.0 (clang-700.0.72), boost /usr/local/Cellar/boost/1.58.0, osx 10.10.5  ....
09:46 < gavinandresen> ... but it is very possible my machine is in some weird state, I upgraded my XCode yesterday.
09:47 < btcdrak> gavinandresen: first rule of fight club - "never upgrade XCode"
09:56 < jgarzik> heh
09:56 < jgarzik> just did last night...
09:59 < cfields> gmaxwell / jgarzik: ping. I must be going crazy. From what I can tell experimentally (and backed by man pages), on Linux, our incoming connections are being treated as blocking.
09:59 < jgarzik> cfields, post-accept(2) you set sockets to non-blocking
09:59 < cfields> because O_NONBLOCK isn't inherited by accept
10:00 < jgarzik> nod
10:00 < cfields> jgarzik: should, sure
10:00 < cfields> but we don't
10:00 < jgarzik> oops
10:00 < jgarzik> seems surprising, seems like it would have been well noticed before now
10:01 < sipa> that would imply that the "opportunistic write" always succeeds
10:01 < cfields> for the most part it should be ok because reads wait for select(), but i'm surprised it hasn't caused issues
10:01 < sipa> that should be trivial to test?
10:01 -!- rubensayshi [~ruben@91.206.81.13] has quit [Remote host closed the connection]
10:01 < jgarzik> no wait - send/recv have per-call NB flags
10:02 < cfields> jgarzik: ah right
10:03 < cfields> jgarzik: still though, i've read about select() acting funny on linux with blocking sockets
10:04 < jgarzik> like haha, that's a laugh funny?
10:05  * jgarzik attempts to implement db iteration without brute force 'select * from table' into RAM ;p
10:06 < cfields> "Under Linux, select() may report a socket file descriptor as "ready for reading", while nevertheless a subsequent read blocks... Thus it may be safer to use O_NONBLOCK on sockets that should not block."
10:06 < jgarzik> recv() will fail in that case
10:06 < cfields> Either way, forcing them to nonblock after accept can't hurt, right?
10:06 < cfields> hmm
10:07 < sipa> jgarzik: i guess you can get some sort of object/state which you can ask for "more results" ?
10:07 < cfields> it wouldn't just block?
10:08 < GitHub132> [bitcoin] MarcoFalke opened pull request #6870: [trivial] Misc cleanup and translations (master...MarcoFalke-2015-trivial3) https://github.com/bitcoin/bitcoin/pull/6870
10:08 < cfields> jgarzik: er, the DONTWAIT. nm.
10:12 < jgarzik> sipa, you can 'select *' and iterate in one direction (forwards; next-result), whereas the CDBIterator wants Next, Prev and Seek operations
10:13  * jgarzik is studying the callers/users of CDBIterator now
10:13 < sipa> jgarzik: i don't think we use prev
10:15 < jgarzik> so it seems
10:16 < sipa> and seek would be equivalent to just starting the search
10:16 < cfields> gavinandresen: that's no good. i'm confused about that error, though
10:17 < jgarzik> sipa, well seek-start, seek-end and seek-key are all different
10:17 < sipa> jgarzik: i think the abstraction can be changed so we have a few 'entry types' (which would translate to separate sqlite tables)
10:17 < sipa> jgarzik: and only have iterators for all-data-within-one-entry-type
10:17 < jgarzik> sure
10:18 < sipa> which would be compatible with both leveldb and sqlite
10:21 < jgarzik> ah a few of these db ops are like Prev(), unused.  job easier++
10:21 -!- murr4y [murray@kvikshaug.no] has joined #bitcoin-core-dev
10:24 < jgarzik> hrm
10:24 < jgarzik> sipa, lex order?
10:25 < jgarzik> seems so
10:43 < sipa> some things need lex order, yes
11:11 -!- Greg__ [~greg@h246.25.40.69.dynamic.ip.windstream.net] has joined #bitcoin-core-dev
11:12 -!- __Greg [~greg@h105.31.40.69.dynamic.ip.windstream.net] has quit [Ping timeout: 264 seconds]
11:28 < gavinandresen> FYI: my build-failing-on-OSX problem went away with a 'git clean -dxf' then re-autogen/configure/build....
11:35 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev
11:36  * jonasschnelli did accidentally sent untracked files to nirvana serval times with `git clean -dxf`...
11:36  * sipa too
11:37 < jonasschnelli> heh
11:37 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 260 seconds]
11:39 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
11:41 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
11:51 -!- jl2012 [~jl2012@unaffiliated/jl2012] has joined #bitcoin-core-dev
11:58 < gmaxwell> I haven't tried it on bitcoin core yet, but the latest functionality in rr (replay debugger) looks pretty great: http://robert.ocallahan.org/2015/10/rr-40-released-with-reverse-execution.html
12:02 < jcorgan> did i hear rumblings about switching to sqlite?
12:03 < maaku> i wish there was a 'git stage --clean' command
12:03 < btcdrak> meeting over at #bitcoin-dev now
12:03 < jcorgan> isn't it git checkout . ?
12:03 < btcdrak> jcorgan: yes, jgarzik is working on it now
12:04 < jcorgan> oh sweet.  then we can use sqlcipher and get full db ondisk encryption
12:04 < jcorgan> it's a drop-in replacement/fork for sqlite that operates at the db page level, like dmcrypte for block devices
12:05 < sipa> why do you need encryption for public data? :)
12:05 < maaku> sipa: i presume he's talking about wallet?
12:05 < sipa> i wouldn't use sqlite for the wallet
12:05 < gmaxwell> jcorgan: you saw that discussion, but I expect it won't happen (At least with sqllite3) as-- if I'm not mistaken-- btcd tried it and found the performance unacceptable.  (but we should anyways, both to have a comparison point, and because sqllite4 might be faster even if 3 is too slow)
12:06 < jcorgan> encrypt ALL THE THINGS
12:07 < gavinandresen> RE: switching to sqlite: this CACM article was very interesting:  http://cacm.acm.org/magazines/2015/10/192379-crash-consistency/abstract
12:07 < gavinandresen> ... bashes on leveldb a bit, praises sqlite ....
12:08  * maaku is very confused as to what the benefit of switching to sqlite would be for this particular use case
12:08 < jgarzik> maaku: it's maintained and reliable
12:09 < gavinandresen> maaku: assuming sqlite is better about crash consistency (and from all I've read, it is), would cut down on the "core sucks, because my PC crashed and then eleven days to reindex..."
12:10 < sipa> yes, that's the only reason
12:10 < sipa> sqlite is known to be rock solid
12:10 < sipa> however, performance would need to be acceptable
12:14 < maaku> sipa: well, also interop and the ease of using a different sql backend
12:16 < jcorgan> i would definitely advocate making a sql abstraction layer with pluggable back ends, with sqlite as the default
12:17 < CodeShark__> I'm using ODB in my stack for that
12:17 < sipa> jcorgan: to be clear, i think SQL is a downside
12:18 < sipa> we're not looking for a database, and turning things into a database is against what we're trying to achieve
12:18 < jgarzik> nod
12:18 < jcorgan> my secret plan revealed, again
12:18 < sipa> our database is not designed to be accessible by multiple processes
12:18 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Quit: AndroIRC - Android IRC Client ( http://www.androirc.com )]
12:18 < sipa> nor do we need any fancy indexes or queries
12:18 -!- CodeShark__ [CodeShark@cpe-76-167-237-202.san.res.rr.com] has quit []
12:18 < gavinandresen> if there's a key-val store as well-supported and rock solid as sqlite..... that's the right answer.
12:18 < jgarzik> I still like the custom db approach, but I'm highly biased
12:18 -!- CodeShark [CodeShark@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
12:19 < sipa> if sqlite offered a mechanism to just use its low-level key-value store, i would
12:19 < sipa> (because it's a database layer built on top of a key-value store)
12:19 < jgarzik> program in sqlite VM assembler and you can...
12:19  * jgarzik runs
12:19 < maaku> imho the answer would be to switch to a better maintained descendent of leveldb (there are many)
12:20 < maaku> also minimizes risk of fork
12:20 < gmaxwell> Its unhelpful to have this conversation in parallel with the meeting.
12:21 -!- bsm1175321 [~bsm117532@38.121.165.30] has joined #bitcoin-core-dev
12:24 -!- luke-jr_ is now known as Luke-Jr
12:26 < kanzure_> upstream maintainers, apparently
12:26 < kanzure_> oops, bad scrollback
12:27 -!- skyraider [uid41097@gateway/web/irccloud.com/x-dnhbflxhkiphansa] has joined #bitcoin-core-dev
12:30 -!- crescendo [~mozart@unaffiliated/crescendo] has joined #bitcoin-core-dev
12:32 < bsm1175321> Don't want to interject in the meeting, but I re-ran the SQLite vs. libdb benchmarks sipa mentioned (because they were 6 years old).
12:32 < bsm1175321> Results here: https://gist.github.com/mcelrath/6952eab246a7c705a0fb
12:35 < kanzure_> huh? why should that not be mentioned?
12:36 < bsm1175321> Because the topic has moved on, and the conclusion was to continue research.
12:39 -!- You're now known as kanzure
12:44 < bsm1175321> Above link updated with 500k records, transactional DB.  (Took a few minutes to run)
12:47 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has joined #bitcoin-core-dev
13:03 < BlueMatt> IIRC gmaxwell was gonna ask the crash consistency guys for their test harness?
13:04 < BlueMatt> so we could test other dbs
13:04 < BlueMatt> or maybe I'm crazy
13:04 < BlueMatt> someone should do that, though
13:04 < BlueMatt> before we decide
13:06 < sipa> maaku: sqlite is better at large writes than leveldb
13:06 < maaku> btcdrak: sqlite4 maybe matches leveldb ... because it's basically using leveldb's datastructure under the hood
13:06 < sipa> as leveldb writes everything twice tovdisk
13:08 < btcdrak> gmaxwell: this was the discussion earlier today about sqlite: https://botbot.me/freenode/bitcoin-core-dev/2015-10-22/?msg=52492157&page=2
13:08 < gmaxwell> btcdrak: yes I read it. I am confident from expirence that its incorrect, but I still think jeff's work is very useful.
13:09 < btcdrak> gmaxwell: yeah, I was also surprised, but fingers crossed something impressive comes out of jeff's research.
13:09 < gmaxwell> (in part because sqllite4 may be faster)
13:09 < btcdrak> sqlite4 is still in development though isnt it?
13:10 < gmaxwell> sure, not useful yet.
13:10 < btcdrak> i wonder how far on they are and if they have a release schedule yet
13:10 < Luke-Jr> am I missing something? why is this being treated as a non-consensus thing?
13:10 < sipa> it obviously is consensus critical
13:10 < sipa> it's not a consensus change though
13:11 < Luke-Jr> we don't and can't know that.
13:11 < sipa> recompiling with a different libc is also consensus critical
13:11 -!- jtimon [~quassel@74.29.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev
13:11 < Luke-Jr> replacing 26kLOC to 140kLOC is impossible to be sure on
13:11 < BlueMatt> Luke-Jr: we should be confident that it is not before anything moves forward
13:11 < sipa> or running on a new linix kernel
13:12 < Luke-Jr> BlueMatt: I don't think we can.
13:12 < helo> i really appreciate the meetings, great idea (btcdrak?)
13:12 < btcdrak> helo
13:12 < sipa> we are making certain assumptions from other software we are using
13:12 < BlueMatt> Luke-Jr: then we're stuck with a shitty db that corrupts randomly and isnt self-consistent (ie not consensus-compatible)
13:12 < Luke-Jr> best I can see happening, is knowing it's unlikely to have any accidentally-triggered failures
13:12 < gmaxwell> Luke-Jr: it's not being treated as anything right now, it's just a test.
13:13 < sipa> that includes compilers, libraries, and operation systems with _well specified_ behaviour
13:13 < gmaxwell> Luke-Jr: by that basis we don't know that leveldb is self-consistent, or consistent between windows / unix, etc.
13:13 < maaku> Other than Windows corruption errors, is this solving any real problems right now?
13:13 < btcdrak> Luke-Jr: research is a good thing.
13:13 -!- tombtc [~tombtc@unaffiliated/tombtc] has joined #bitcoin-core-dev
13:13 < gmaxwell> maaku: we have durability problems on all platforms, though the windows ones are very severe. 
13:13 < Luke-Jr> btcdrak: yes, if this is just research fine
13:14 < gmaxwell> The performance of leveldb is also problematic already, (well reported for very large databases), enough that our fat caching layer is very very criticial for performance, and thats causing us other problems.
13:14 < gmaxwell> maaku: durability problems in this database are incompatible with the survival of a network where puning is widely used.
13:14 < sipa> puny network
13:15 < gmaxwell> hah
13:15 < gmaxwell> pruning*
13:15 < btcdrak> LOL
13:16 < wumpus> in general leveldb works fine; greatest concern is that it seems to be no longer maintained, so any bugs (like corruption on crashes in windows) is unlikely to get solved
13:16 < maaku> i thought gmaxwell was declaring sipa a risk to bitcoin there
13:17 < gmaxwell> maaku: well that much is obvious.
13:17 < sipa> we couls try to find another win env for leveldb
13:17 < sipa> there may be bugs in that
13:18 < jonasschnelli> i know this sounds hard: but did we never consider to stop supporting windows? Would this be so wrong?
13:18 < wumpus> I'm sure it is possible to fix the windows issue, but it's not really nice to inherit a dependency and suddenly have to maintain it
13:19 < Luke-Jr> jonasschnelli: I think it would be problematic to do so right now.
13:19 < Luke-Jr> jonasschnelli: as bad as it is, many people still use it
13:19 < gmaxwell> jonasschnelli: I think it would be a bad idea, its a very widely used platform and dropping it would be a major move against the ability of people to independnantly run, audit, etc. the bitcoin system.
13:19 < wumpus> jonasschnelli: well I've been close to declaring that. it wouldn't be pretty but we have zero active developers for windows.
13:20 < gmaxwell> The security problems of windows stink, but hardware wallets are a tidy tool to address that.
13:20 < wumpus> jonasschnelli: then again, the software is working for a lot of people also on windows
13:20 < jonasschnelli> not even speaking for the wallet. Just the node/core itself.
13:20 < gmaxwell> we are in a state right now where it is is almost constructively non-supported by our failure to do a a great job about it.
13:20 < maaku> well to be clear, gmaxwell's concerns about a pruned network wouldn't apply if the durability issues were wholly isolated to the windows platform
13:20 < Luke-Jr> jonasschnelli: then people will just stop running a node :/
13:21 < wumpus> gmaxwell: I disagree
13:21 < Luke-Jr> before we drop Windows support, we would ideally want the alternative to be "run a node on Linux", not "stop running a node altogether"
13:21 < gmaxwell> maaku: they are not wholly isolated to windows, though perhaps its safe enough elsewhere that my concern is less of an issue.
13:21 < jonasschnelli> wumpus: Agree. It's running. But i think we should not move away form leveldb because of windows... 
13:21 < wumpus> gmaxwell: many people are running it on windows, and not complaining at all
13:21 < wumpus> gmaxwell: no need to blow this issue out of proportion
13:21 < gmaxwell> wumpus: and many people have stopped running it too.
13:21 < wumpus> gmaxwell: obviously
13:22 < gmaxwell> I believe more people have stopped running bitcoin core on windows than currently run it.
13:22 < gmaxwell> (stopped due to corruption issues of varrious forms)
13:22 < wumpus> well this is an open source project - if we don't have developers supporting windows, we can't support windows
13:22 < jonasschnelli> are most corruption issues caused by leveldbs layer?
13:22 < gmaxwell> (not just leveldb but also AV)
13:22 < sipa> jonasschnelli: we don't know
13:23 < gmaxwell> We know there is leveldb corruption on unclean shutdown, and we know about antivirus mediated corruption. But there may be more things we don't know about.
13:23 < wumpus> if the corruption issues are as bad as you say they are, and no one is fixing them,  then we should indeed drop windows support
13:23 < Luke-Jr> maybe if we stop supporting Windows "officially", but leave it open, others will step up to do it
13:23 < wumpus> but I don't believe you
13:23 < gmaxwell> wumpus: or we should get more windows developers.
13:23 < wumpus> gmaxwell: well, good luck with that
13:23 < gmaxwell> Which can be done.
13:24 < gmaxwell> wumpus: I mean I can _hire_ people for this, but it hasn't been on my radar.
13:24 < wumpus> in any case, this is not constructive *goes back to sleep*
13:24 < jonasschnelli> we could ship it together with a vmbox ubuntu. *duck*
13:24  * Luke-Jr wonders if Diapolo is good enough at C++ that he could transition to more than just GUI if he gets funding
13:24  * jonasschnelli looks strange at Luke-Jr
13:24 < gmaxwell> wumpus: :-/ I think you're misunderstanding my perspective, sorry I've communicated poorly.
13:24 < Luke-Jr> jonasschnelli: got a better suggestion? :P
13:25 < Luke-Jr> I don't know many Windows developers..
13:25 < jonasschnelli> Luke-Jr: hah. No. Not really. :)
13:25 < jonasschnelli> serious: what would be the overhead to run bitcoin-qt/core in a vm on windows?
13:25 < Luke-Jr> maybe the people who did those libconsensus bindings?
13:26 < wumpus> gmaxwell: it's not impossible to just solve the windows crash corruption issue
13:26 < jonasschnelli> 10% CPU loss, 512MB ram overhead?
13:26 < Luke-Jr> jonasschnelli: I would be surprised if a VM kill reproduced the problem
13:26 < wumpus> gmaxwell: I even proposed to help with it today if someone can send me a corrupted database
13:26 < gmaxwell> wumpus: yup. did anyone do that?
13:27 < wumpus> gmaxwell: no
13:27 < jonasschnelli> Luke-Jr: the idea would be to bundle bitcoin-qt/core on windows together with vbox and a tiny distro. This would eliminate some ugly platform dependents.
13:27 < Luke-Jr> gmaxwell: I tried to, but couldn't reproduce it with IBD, and shortly thereafter my USB Armory died entirely. So it seems likely my problem was a bad microSD
13:27 < Luke-Jr> jonasschnelli: that sounds like a terrible UX
13:28 < jonasschnelli> Luke-Jr: the UX could still be native (once it's decoupled)
13:28 < wumpus> Luke-Jr: yes, your problem is likely adifferent one. You're the only one reporting it on linu
13:28 < wumpus> jonasschnelli: if you go that far to make a bundle you may as well fix the windows version :-) (it would be about as much development work)
13:29 < gmaxwell> jonasschnelli: doesn't jive really well with the resource costs of running a node.... VM with the constantly expanding storage is no fun. :P plus overheads. :P
13:29 < Luke-Jr> does UML support Windows? ;)
13:29 < gmaxwell> Fixing the windows shouldn't be a big deal once we get enough repros and data, similar to OSX.
13:29 < gmaxwell> sounded like it might be the same issue.
13:30 < wumpus> yeah, let's just solve the database problem instead of trying to work around it
13:30 < wumpus> even if we have to put a flush after every line in leveldb ...
13:30 < gmaxwell> I considered the exploration of other options to be orthorgonal with windows being on fire FWIW.
13:30 < jonasschnelli> gmaxwell: nowadays VM overhead is tiny... it's not perfect. But better a solution that works with a lost of 10-20% in overhead than unsolved corrupted databases.
13:30 < wumpus> gmaxwell: sure, I still like the idea of exploring other databases
13:31 < wumpus> but yes, orthogonally
13:31 < sipa> i really like the idea of just being able to use a maintained and tested database
13:31 < gmaxwell> (also the attempt may turn up bugs elsewhere in our codebase)
13:31 < wumpus> don't we all
13:31 < sipa> and not a hack we had to pull together
13:31  * wumpus thought leveldb was that
13:31 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Quit: Leaving]
13:31 < gavinandresen> a bug bounty worked for the last leveldb corruption issue we had (if i recall correctly). I'm still holding some bitcoin in the core dev expenses fund
13:31 < sipa> wumpus: leveldb windows certainly isn't
13:32 < Luke-Jr> jonasschnelli: if someone really wants to investigate VM stuff, I'd suggest instead a thin OS that runs *Windows* in the VM, and provides a hardwarewallet-like interface on top
13:32 < jonasschnelli> is there an approach to test sqlites performance which represents our db-style?
13:32 < sipa> wumpus: we have local modifications to the win env
13:32 < wumpus> jonasschnelli: best would be to just try it
13:32 < gmaxwell> jonasschnelli: yes, testing bitcoin core using <alternative database> :P  it shouldn't be that much work; presumably jeff will report back on that soon
13:33 < jonasschnelli> sqlite could initially be slower. But it's better maintained and much more portable.
13:33 < sipa> jonasschnelli: there is slower and slower
13:33 < jonasschnelli> Indeed
13:33 < sipa> if it's slower for things our caches compensates for, who cares
13:33 < gmaxwell> jonasschnelli: speed is a security consideration for us currently, if you were talking about 5% or something I'd agree. but thats now what I expect.
13:34 < gmaxwell> s/now/not/
13:34 < sipa> if it's slower to the point that it affects block propagation, it's a no go
13:34 < gmaxwell> s/currently/sadly/
13:34  * wumpus doubts it will make much of a difference
13:34 < jonasschnelli> Quote from sqlite4: The default built-in storage engine is a log-structured merge database. It is very fast, faster than LevelDB, supports nested transactions, and stores all content in a single disk file. Future versions of SQLite4 might also include a built-in B-Tree storage engine.
13:34 < jonasschnelli> promising. :)
13:35 < wumpus> there are so many factors influencing performance of bitcoind, that a slightly slower database won't be a big issue
13:35 < sipa> yes, sqlite4 sounds awesome _once_ it has had the same amount of testing and battle hardending as sqlite3
13:35 < wumpus> jonasschnelli: yes that was pasted before today :)
13:35 < jonasschnelli> ha.. okay. I'd love to play around with it. But my stack of things to work down is just to big right now.
13:36 < gmaxwell> we might also want to maintain a patch for sqllite4 and use it to try to get the sqllite devs to use us as a test harness. :P
13:36 < jonasschnelli> sipa: sqlite is widely used, also in almost every browser (local dbs) and smartphone (android and iOS IIRC). They very likely to update once to sqlite4... 
13:36 -!- Greg__ [~greg@h246.25.40.69.dynamic.ip.windstream.net] has left #bitcoin-core-dev []
13:38 < sipa> but quote from one of the sqlite devs a few months ago "sqlite4 is a dev toy"
13:38 < jonasschnelli> sipa: that sounds perfect for our bitcoin <1.0 version. :)
13:39 < sipa> jonasschnelli: if only we didn't have this pesky economy thing that relies on bitcoin
13:39 < jonasschnelli> sipa: na... that are just some fiat exchange bubbles.. :)
13:39 < gmaxwell> Lets fix that first then. :P
13:40 < wumpus> trying it out doesn't hurt, no one is talking about releasing with it...
13:40 < sipa> wumpus: exactly
13:40 < sipa> dev toy :p
13:40 < jonasschnelli> wumpus: +1 ... and a such pr would at least take 1/2 year to get in.
13:40 < gmaxwell> well as I said, might even be possible to get the sqllite developers to test using us. We're a pretty cool load generator (esp with signatures off)
13:41 < sipa> gmaxwell: we're boring
13:41 < sipa> single-threaded access, only bulk writes and small random reads
13:48 -!- tombtc [~tombtc@unaffiliated/tombtc] has quit [Quit: quit]
14:10 < jcorgan> < Luke-Jr> jonasschnelli: if someone really wants to investigate VM stuff, I'd suggest instead a thin OS that runs *Windows* in the VM
14:11 < jcorgan> *cough* Qubes OS *cough*
14:11 < Luke-Jr> jcorgan: Qubes OS is not very thin, and doesn't do GPU passthrough
14:12 < jcorgan> didn't think about GPU
14:12 < jcorgan> but thinness is poorly defined
14:12 < btcdrak> sipa: are the tests in #6816 (versionbits) enough? I was thinking it would be good to have some rpc-tests there as well, generating blocks and running through a couple of scenarios.
14:13 < btcdrak> or is that overkill?
14:13 -!- jl2012 [~jl2012@unaffiliated/jl2012] has quit [Ping timeout: 260 seconds]
14:21 < CodeShark> btcdrak: I think we should probably do some regtests with the integration, not necessarily over RPC
14:21 < CodeShark> well...
14:21 < CodeShark> other than just generating blocks
14:21 < CodeShark> I guess just calling generate and getblockchaininfo
14:26 < btcdrak> CodeShark: I mean tests like https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/bipdersig.py etc
14:27 < jgarzik> It builds: https://github.com/jgarzik/bitcoin/tree/2015_sqlite
14:28 < btcdrak> jgarzik: That was fast
14:28 < CodeShark> jgarzik: nicely done :)
14:28 < sipa> yes, but does it run linyx?
14:30 < CodeShark> btcdrak: the -blockversion thing will probably not be a good idea anymore using versionbits
14:30 < CodeShark> it will be better to provide a list of BIPs you don't want to set the bit for
14:30 < jgarzik> a few bits can be forwarded upstream immediately, like Seek* and Prev() removal
14:31 < btcdrak> CodeShark: well obviously the bipdersig.py tests are for an ISM sf.
14:32 < btcdrak> CodeShark: I'm just saying a set of tests like that which generate blocks and simulate a sf rollout but using versionbits protocol.
14:32 < CodeShark> I suppose we can just use VERSION_HIGH_BITS and VERSION_HIGH_BITS | 0x1 instead of versions 2 and 3 :)
14:33 < Luke-Jr> CodeShark: does versionbits sanitise the encoding btw? ;)
14:33 < Luke-Jr> ie, define the first 32-bits to be a big endian number
14:33 < maaku> Luke-Jr: it's little endian
14:33 < Luke-Jr> would be nice to fix that at the same time
14:33 < CodeShark> encoding? versionbits doesn't deal with serialization stuff
14:33 < btcdrak> CodeShark: it makes the most conclusive tests that it works, and provides some protection against regressions.
14:34 < maaku> Luke-Jr: why? nothing else is big endian
14:34 < Luke-Jr> CodeShark: anything consensus-relevant must deal with encoding
14:34 < maaku> Luke-Jr: he means versionbits doesn't touch serialization
14:34 < CodeShark> Luke-Jr: versionbits deals with block header data that has already been deserialized
14:34 < Luke-Jr> maaku: big endian is standard for protocols; and hashes at least are
14:34 < sipa> Luke-Jr: please
14:35 < sipa> we're not changing the serialization of block headers
14:36 < jgarzik> big endian is dead
14:37 < sipa> no endianness flamewar please
14:37 < CodeShark> in any case, the serialization format is entirely a separate issue...versionbits will play no part in that either way
14:39 < Luke-Jr> so the bits are numbered 7 6 5 4 3 2 1 15 14 13 12 11 10 9 8 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24 ?
14:39 < sipa> irrelevant
14:39 < Luke-Jr> …
14:39 < CodeShark> versionbits deals with ints, uint32_t, etc...
14:39 < Luke-Jr> it's relevant because if I set the wrong BIP, it won't work.
14:40 < sipa> the block header serialization defines nVersion as a 32-bit little-endian signed integer
14:40 -!- aj [aj@cerulean.erisian.com.au] has joined #bitcoin-core-dev
14:40 < Luke-Jr> wrong bit*
14:40 < CodeShark> it doesn't care what the underlying representation is for the particular architecture
14:40 < sipa> versionbits changes the semantics of the nVersion integer
14:40 < CodeShark> by the time versionbits comes into play, the version field has already been decoded into an int
14:41 < Luke-Jr> CodeShark: that is an implementation detail. the whole point of version bits is that it is no longer an int, but a bit array
14:41 < CodeShark> to test a bit, you do (nVersion >> bit) & 0x1
14:41 < sipa> Luke-Jr: it's still an int; an int in which particular bits are set
14:41  * Luke-Jr sighs.
14:42 < sipa> now please stop this silly discussion, we're not redefining how integers work
14:42 < gmaxwell> but, base-3!!!
14:42 < Luke-Jr> fine, not worth the time to argue about this stupid design decision.
14:43 -!- sipa [~pw@2a02:348:86:3011::1] has left #bitcoin-core-dev []
14:45 < CodeShark> base-3 would allow us to do yea/nay/abstain :p
14:46 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
14:46 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has joined #bitcoin-core-dev
14:52 < Luke-Jr> the annoying part will be having to answer questions in a few years why it's misordered such. like I had to do with the getwork data for years
14:55 < phantomcircuit> jgarzik, you have benchmarks for sqlite?
14:55 < phantomcircuit> my experience using it in the past has been that it's slow as hell for inserts or any kind of concurrent access
14:56 < phantomcircuit> probably time would be better spent on building an interface spec for databases
14:58 < CodeShark> for insertions we could just use a raw file - the blockchain is a linear structure anyhow. then all we would need to do in sqlite is keep track of file positions ;)
15:03 < phantomcircuit> CodeShark, utxo only
15:04 < CodeShark> yeah, so we maintain a utxo index
15:04 < CodeShark> or are you talking about a pruned node?
15:04 < phantomcircuit> the index would be roughly the same size as the utxo data :P
15:04 < CodeShark> lol
15:05 < CodeShark> ok, for a utxo database perhaps it doesn't make so much sense
15:05 < phantomcircuit> yeah the actual data is really small
15:06 < gmaxwell> CodeShark: txindex works like you suggest there.
15:07 < gmaxwell> but for utxo set we get benefits from reducing the working set. saving 20 bytes of scriptpubkey/value in exchange for another random seek to the middle of some huge block file where 2/3rd of the data is not utxo relevant, basically defeates disk caching entirely. :)
15:12 < GitHub160> [bitcoin] petertodd opened pull request #6871: Full-RBF with opt-out (master...2015-10-rbf-with-opt-out) https://github.com/bitcoin/bitcoin/pull/6871
15:22 -!- skyraider [uid41097@gateway/web/irccloud.com/x-dnhbflxhkiphansa] has quit [Quit: Connection closed for inactivity]
15:33 < bsm1175321> G
15:51 -!- bsm1175321 [~bsm117532@38.121.165.30] has quit [Quit: Leaving.]
15:51 -!- bsm1175321 [~bsm117532@38.121.165.30] has joined #bitcoin-core-dev
15:53 < GitHub51> [bitcoin] TheBlueMatt opened pull request #6872: Remove UTXO cache entries when the tx they were added for is removed/does not enter mempool (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6872
15:53 -!- sipa [~pw@2a02:348:86:3011::1] has joined #bitcoin-core-dev
16:00 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
16:05 -!- dcousens [~anon@d110-33-203-106.mas801.nsw.optusnet.com.au] has joined #bitcoin-core-dev
16:05 < dcousens> petertodd: I hope you don't mind my questions :)
16:06 < phantomcircuit> BlueMatt, the last commit on #6872 seems to evict from the cache things which other transactions in the mempool want cached
16:06 < phantomcircuit> ie AcceptToMempool fails because it's a double spend of something with a higher feerate
16:08 < petertodd> dcousens: no worries!
16:08 < phantomcircuit> scratch that
16:08 < phantomcircuit> i missed the check
16:08 < petertodd> dcousens: I'm doing three things at once so the responses are probably sounding a bit terse :)
16:08 < BlueMatt> phantomcircuit: ahh, ok, good, i was about to say
16:08 < dcousens> petertodd: all good, I read everything in good spirit, just hope it comes across the same :)
16:18 -!- molly [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
16:21 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 260 seconds]
16:40 -!- sipa [~pw@2a02:348:86:3011::1] has quit [Ping timeout: 256 seconds]
16:41 -!- dcousens [~anon@d110-33-203-106.mas801.nsw.optusnet.com.au] has quit [Ping timeout: 268 seconds]
16:42 -!- sipa [~pw@2a02:348:86:3011::1] has joined #bitcoin-core-dev
16:56 -!- sipa [~pw@2a02:348:86:3011::1] has quit [Changing host]
16:56 -!- sipa [~pw@unaffiliated/sipa1024] has joined #bitcoin-core-dev
17:05 -!- dcousens [~anon@203-206-217-234.perm.iinet.net.au] has joined #bitcoin-core-dev
17:10 -!- challisto [~challisto@unaffiliated/challisto] has quit [Quit: Leaving]
17:36 -!- PRab [~chatzilla@2601:40a:8000:8f9b:bd13:f19b:c988:ff92] has quit [Read error: Connection reset by peer]
17:54 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-pkozjheyudppxmcp] has quit [Quit: Connection closed for inactivity]
18:18 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Quit: Leaving]
18:18 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev
18:25 -!- dcousens [~anon@203-206-217-234.perm.iinet.net.au] has quit [Ping timeout: 240 seconds]
18:37 -!- dcousens [~anon@203-206-217-234.perm.iinet.net.au] has joined #bitcoin-core-dev
18:38 < dcousens> petertodd: maybe change the PR title to `Full-RBF with nSequence opt-out`?
18:38 < dcousens> petertodd: it reads as though its an opt-out for the node (until you read the PR itself, obviously)
18:38 < dcousens> (but even still, I had to check the code to be sure)
18:41 < GitHub159> [bitcoin] jgarzik opened pull request #6873: [cleanup] leveldbwrapper becomes more generic 'dbwrapper' module (master...2015_dbwrapper) https://github.com/bitcoin/bitcoin/pull/6873
18:42 < gmaxwell> It would also be less confusing if it were called opt-in. (since transactions are currently opt out under the design)
18:42 < dcousens> gmaxwell: agreed
18:42 < dcousens> though it is called opt-in in the commit
18:42 < dcousens> So, maybe consistency there
18:48 < dcousens> jgarzik: in 6873
18:48 < dcousens> what actually makes CDBWrapper agnostic of leveldb?
18:48 < dcousens> It seems to still use all the statuscode types
18:48 < dcousens> And the wrapper can still be instantiated by itself
18:49 < dcousens> Or is this just a transitionary step?
18:49 < sipa> dcousens: it's API-agnostic... code using it doesn't need to know that it's backed by leveldb
18:50 < sipa> even though you can't compile it any other way
18:50 < jgarzik> nod.  The interface is backend agnostic
18:50 < dcousens> jgarzik: then maybe the PR should be about removal of leveldb types in DbWrapper
18:51 < jgarzik> dcousens, did you look at the sqlite commit?
18:51 < dcousens> sqlite commit?
18:51 < dcousens> nvm
18:51 < dcousens> right
18:52 < dcousens> cheers
18:52 < dcousens> jgarzik: my bad for not reading the PR before reading the changes
18:53 < dcousens> this just reads weird: https://github.com/bitcoin/bitcoin/pull/6873/files#diff-ac09ab7f710fc2fb7850f59601545636R137
18:53 < dcousens> Given its called a 'wrapper', but its now generic,  seems a bit odd IMHO
18:54 < sipa> call it DbInterface then
18:54 < phantomcircuit> jgarzik, :)
18:54 < dcousens> sipa: but its not just an interface no,  it instantiates and manages the entire thing right?
18:55 < dcousens> its more like 'Db', ha
18:55 < jgarzik> it is a class that wraps the backend database in a nice snuggly generic interface
18:56 < gmaxwell> (1) dear god, are people really arguing about this, (2) dbwrapper says what it does!
18:56 < jcorgan> jgarzik: compile ack on the 2015_sqlite branch, but looks like retains all the leveldb naming in the build system, checking now
18:57 < dcousens> gmaxwell: wasn't arguing, just seemed better to ask here than ask about it on GH seeing as its probably me just missing context
18:57 < jgarzik> jcorgan, naming?  2015_sqlite branch should not include any leveldb naming in the bitcoin->sqlite path.  it _does_ build leveldb.  it _does not_ link leveldb to bitcoind.
18:58 < jcorgan> oh, that's what i'm seeing then
18:58 < sipa> dcousens: then DbWrapper is a better name! it's something that wraps a db, but you don't know or care what kind of database that is
18:59 < dcousens> gmaxwell: in any case, a wrapper typically wraps an existing object no?
18:59 < dcousens> in this case, the object is entirely owned by the wrapper
18:59 < sipa> grrrrr, EncapsulatedDb then?
18:59 -!- gmaxwell [greg@wikimedia/KatWalsh/x-0001] has left #bitcoin-core-dev []
18:59 < dcousens> haha, sipa this isn't important, I'm just curious
19:01 < jcorgan> jgarzik: there are some stray leveldb name issues
19:01 < jgarzik> jcorgan, where?
19:01 < jcorgan> like using 'leveldb_error' in dbwrapper.cpp
19:01 < jgarzik> hrm, I thought I got that one :(
19:02 < jgarzik> jcorgan, no, wait, I did.  git pull.  :)
19:02 < jcorgan> it's still defined in dbwrapper.h, try commenting that and see where the errors
19:02 < dcousens> sipa: yeah, quick investigation of the wrapper/adapter pattern is that it is typically applied as a glue code, aka, between two things with their own ownership.
19:02 < dcousens> In this case, its not that, so, anyway.  Just food for though
19:02 < dcousens> thought*
19:03 < dcousens> tl;dr EncapsulatedDb would probably be better
19:03 < jcorgan> i'm using b778481
19:03 < jgarzik> jcorgan, git pull
19:03 < CodeShark> HighlyVerboseButGenericNameForDbInterface
19:04 < jgarzik> dbwrapper is just fine, let's move on.
19:04 < sipa> yes, please
19:04 < jcorgan> jgarzik: ok, that fixed it
19:04 < dcousens> jgarzik: well, it causes confusion in anyone who hasn't read through that part of the source.  So, IMHO it matters enough to be discussed.  But not to block your PR.  Moving on :)
19:05 < jcorgan> but to be pedantic, there are a bunch of 'leveldb' strings in the comments :-)
19:06 -!- arowser [~arowser@106.120.101.38] has joined #bitcoin-core-dev
19:06  * jgarzik is focusing on operating stuff, like why this unit test doesn't want to compile my sql
19:06 < jgarzik> *operational
19:06 < jcorgan> no worries.  i haven't tested anything, but compiles fine on ubuntu trusty
19:07 < jcorgan> and 6873 is fine
19:07 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving]
19:32 < GitHub80> [bitcoin] pstratem opened pull request #6874: Net: Cork socket send writes with MSG_MORE (master...msg_more) https://github.com/bitcoin/bitcoin/pull/6874
19:32 < phantomcircuit> jgarzik, it would be nice if there was a virtual base class
19:32 < phantomcircuit> DBWrapperInterface
19:33  * phantomcircuit goes to look for some InterfaceFactories
19:37 < dcousens> phantomcircuit: if the default is being hard-coded in at each entry anyway
19:37 < dcousens> probably not much point
19:38 < dcousens> unless the wrapper is instantiated at a much earlier point in the program, and passed through.  At which point then the generic interface could be used
19:38 < dcousens> s/earlier/higher
19:38  * jgarzik reminds channel: we've moved on
19:41 < phantomcircuit> dcousens, well it's only actually named in one place i think so it doesn't much matter
19:52 < GitHub95> [bitcoin] TheBlueMatt opened pull request #6875: Fix pre-push-hook regexes (master...verify-commits-fixes) https://github.com/bitcoin/bitcoin/pull/6875
19:54 < GitHub11> [bitcoin] TheBlueMatt closed pull request #6875: Fix pre-push-hook regexes (master...verify-commits-fixes) https://github.com/bitcoin/bitcoin/pull/6875
19:58 -!- PRab [~chatzilla@2601:40a:8000:8f9b:bd13:f19b:c988:ff92] has joined #bitcoin-core-dev
20:37 < phantomcircuit> jgarzik, updated 6874
21:24 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Ping timeout: 272 seconds]
21:24 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev
21:26 < phantomcircuit> <phantomcircuit> jgarzik, updated 6874
21:51 -!- jl2012 [~jl2012@unaffiliated/jl2012] has joined #bitcoin-core-dev
22:13 -!- AtashiCon [arnavion@unaffiliated/arnavion] has quit [Quit: AtashiCon]
22:18 -!- AtashiCon [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev
22:21 < phantomcircuit> possibly we should at least try to reconnect to disconnected peers...
22:26 -!- maaku [~quassel@botbot.xen.prgmr.com] has quit [Remote host closed the connection]
22:28 -!- maaku [~quassel@botbot.xen.prgmr.com] has joined #bitcoin-core-dev
22:29 -!- evoskuil [~evoskuil@c-73-225-134-208.hsd1.wa.comcast.net] has joined #bitcoin-core-dev
23:23 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has joined #bitcoin-core-dev
23:28 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.]
23:33 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-wmoudgivcgjupqbj] has joined #bitcoin-core-dev
23:34 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
23:35 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Client Quit]
23:37 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
23:54 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has joined #bitcoin-core-dev
23:54 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has quit [Client Quit]