--- Day changed Sat Oct 24 2015 00:11 -!- danielsocials [~quassel@221.220.232.153] has joined #bitcoin-core-dev 00:16 -!- danielsocials_ [~quassel@222.130.179.179] has joined #bitcoin-core-dev 00:17 -!- danielsocials [~quassel@221.220.232.153] has quit [Ping timeout: 240 seconds] 00:27 -!- GAit [~GAit@2-228-102-100.ip191.fastwebnet.it] has quit [Quit: It was good while it lasted!] 00:40 -!- ParadoxSpiral_ [~ParadoxSp@p508B9F77.dip0.t-ipconnect.de] has joined #bitcoin-core-dev 00:43 -!- ParadoxSpiral [~ParadoxSp@p508B9986.dip0.t-ipconnect.de] has quit [Ping timeout: 240 seconds] 00:58 < jgarzik> jcorgan, btcdrak: 2015_sqlite branch now works, passes tests 00:58 < jgarzik> not yet performance-tuned 01:00 < wumpus> great 01:00 * wumpus switches to sqlite 01:06 < jgarzik> no apparent transaction size limit. "I am able to insert 10 million rows in a single transaction" 01:07 < jgarzik> BEGIN...COMMIT maps easily to DBWrapper's batches 01:21 < wumpus> that's good news 01:22 < wumpus> one question: will using the mysql branch blow my old leveldb database, or can they exist side by side? 01:23 < wumpus> will just create a new datadir to be sure 01:25 < jgarzik> should be able to exist side by side 01:25 < jgarzik> sqlite database is a file named "db" 01:26 < jgarzik> but don't trust me - be certain and create a new datadir :) 01:27 < jgarzik> 1) Here are all the tweaks for an sqlite database: https://www.sqlite.org/pragma.html 01:28 < jgarzik> 2) git pull the latest 2015_sqlite branch, it includes some key performance tweaks in dbwrapper.cpp (grep for PRAGMA) 01:28 * jgarzik heads back to bed *poof* 01:30 < btcdrak> jgarzik: I'll take a look 01:30 < GitHub65> [bitcoin] giacecco opened pull request #6885: Instructions on how to make the Homebrew OpenSSL headers visible... (master...master) https://github.com/bitcoin/bitcoin/pull/6885 01:31 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-iexcitcqxjxehrvv] has joined #bitcoin-core-dev 01:35 -!- danielsocials_ [~quassel@222.130.179.179] has quit [Ping timeout: 260 seconds] 01:45 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 05:47 < wumpus> 2015-10-24 12:45:29 UpdateTip: new best=000000000000000bf325356179fb8876fe40e250c9e31082242f70f89ecbcd0b height=240923 log2_work=70.308629 tx=1 05:47 < wumpus> 9292749 date=2013-06-11 12:51:49 progress=0.093959 cache=65.5MiB(34687tx) 05:47 < wumpus> 2015-10-24 12:46:10 UpdateTip: new best=00000000000000ecca86ba925835b0909eeba33fd90ae9858c01e088d4d13bcf height=240924 log2_work=70.308695 tx=1 05:47 < wumpus> 9293107 date=2013-06-11 12:59:51 progress=0.093961 cache=2.0MiB(0tx) 05:48 < wumpus> it seems like the flushing takes quite a long time with sqlite (40 seconds in this case), haven't done a direct comparison with leveldb though 05:48 < wumpus> but it blazes past N blocks, then hangs noticably on the flush, longer than I remember 05:50 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev 06:00 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 06:03 -!- molly [~molly@unaffiliated/molly] has quit [Ping timeout: 260 seconds] 06:04 < wumpus> at first glance it seems to do a lot of calls to fsync() 06:04 < wumpus> (unscientifically tested by running in gdb and breaking+backtracing a few times) 06:35 -!- danielsocials [~quassel@222.130.179.179] has joined #bitcoin-core-dev 06:41 < wumpus> yes it's fsyncing - removing the fsync calls (obviously a stupid idea in itself) flush time is down to <10 seconds. I understand calling fsync, but is it doing so for every single statement? 07:06 -!- danielsocials [~quassel@222.130.179.179] has quit [Ping timeout: 256 seconds] 07:15 < jcorgan> wumpus: confirming same behavior here 07:19 < wumpus> "PRAGMA synchronous=0" works too (make sure it's executed every time the database is opened) instead of commenting out fsyncs - though won't be very resistant to crashes https://www.sqlite.org/pragma.html#pragma_synchronous 07:20 < jcorgan> heh, isn't the whole reason we're testing this is to improve crash resistance? :-) 07:21 < wumpus> yeah... 07:22 < jcorgan> once this reindexes i'll bet steady-state performance won't really be any different 07:22 < wumpus> so sqlite's idea of crash resistance seems to be 'fsync after every file operation'. Or maybe it's after some buffer fills that can be increased, I don't know. 07:22 < wumpus> jcorgan: I'll bet the same 07:23 < jcorgan> hmm, reindexing is actually speeding up as it goes along 07:25 < jcorgan> eyeballing it it is about 150-200 blocks/sec, with peak disk writes of about 70-80 MB/sec 07:26 < wumpus> jgarzik: the pragma tweaks in sql_db_init are not persistent - they need to be done every time the db opens, not only when it is created 07:28 -!- danielsocials [~quassel@222.130.179.179] has joined #bitcoin-core-dev 07:31 < jgarzik> wumpus, very odd - I would think page size is persistent 07:32 < wumpus> this sounds interesting: https://www.sqlite.org/wal.html 07:32 < jgarzik> nod 07:32 < jgarzik> Plenty of tips (sounds like you already figured out some) in https://www.sqlite.org/cvstrac/wiki?p=PerformanceTuning 07:33 < jgarzik> you can play around with logging types 07:33 < wumpus> jgarzik: probably the database is created with one page size, but e.g. cache size isn't remembered 07:37 < wumpus> good news, PRAGMA journal_mode=WAL; seems to solve the excessive-fsync problem too 07:39 < wumpus> "WAL works best with smaller transactions. WAL does not work well for very large transactions. For transactions larger than about 100 megabytes, traditional rollback journal modes will likely be faster" that's strange - we certainly have transactions that big 07:39 < jgarzik> In C++, is there a one-line way to convert an int to a std::string ? 07:39 < jgarzik> i.e. "foo" + numstr(22) 07:39 < wumpus> we have itostr in utilstrencodings.h 07:40 < wumpus> and i64tostr 07:40 < jgarzik> ok, perfect 07:42 -!- fkhan [weechat@gateway/vpn/mullvad/x-pafengwxecqdlpgq] has quit [Ping timeout: 252 seconds] 07:44 < jgarzik> wumpus, pushed the db init fix to 2015_sqlite (where params like cache_size are configured every time, but 'create table' happens only once) 07:45 < jgarzik> wumpus, WAL should be good for us 07:45 < jcorgan> is that in your update? 07:45 < jgarzik> jcorgan, WAL? no 07:47 < jgarzik> jcorgan, it's pretty obvious where to add configuration lines at the top of dbwrapper.cpp, so it's straightforward 07:47 < jgarzik> Another todo item is making the cache size configurable 07:48 < jgarzik> (via runtime GetArg, I mean) 07:48 -!- fkhan [weechat@gateway/vpn/mullvad/x-rymbotvculqltgnb] has joined #bitcoin-core-dev 07:50 < jgarzik> 70-80 MB/sec is pretty darned good 07:50 < jcorgan> btrfs on hardware raid 10 :-) 07:50 < jgarzik> I would be interested in the wall clock reindex time 07:50 < jgarzik> of master vs sqlite 07:51 < jcorgan> i'm restarting with WAL, i'll time it 07:51 < jcorgan> btrfs snapshots are the bomb 07:52 < wumpus> with WAL, PRAGMA wal_autocheckpoint has a lot of influence in the number of fsyncs 07:53 * jgarzik doesn't see a need for fsync inside a batch 07:54 < jgarzik> as long as post-crash we see a consistent picture, we can lose the WAL-in-progress 07:54 < wumpus> well it *looks* to me that it's this: during a batch it writes to the journal, and it fsyncs the journal. I agree though. 07:54 < jgarzik> changes are batched to std::vector<> internally, and then flooded to the db in a rapid BEGIN..INSERT*..COMMIT sequence. 07:55 < wumpus> no need to checkpoint *ever* during a batch 07:55 < jgarzik> correct 07:56 < sipa> cache size is controlled by -dbcache already 07:56 < sipa> part of it is assigned to pcoinsTip cache, part to the database layer itself 07:57 < sipa> using a totally arbitrary formula 07:57 < jgarzik> sipa, the specific need is for sqlite to move from static constant stored within a constant "foo" string to GetArg configured with that 07:58 < jgarzik> sipa, Line 23 of https://github.com/jgarzik/bitcoin/blob/4d2e72900de85a1e2ffbc9470df05794242b82b9/src/dbwrapper.cpp#L23 08:04 < sipa> oh, yes! 08:06 < sipa> i'm just saying, no need for a new GetArg, there is already logic for this in init.cpp 08:07 < jgarzik> yep 08:07 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.] 08:10 < sipa> note that the chainstate only really has a durability requirement when pruning 08:10 < sipa> otherwise, any old but consistent state is acceptable 08:11 < sipa> though blockindex flushes are hard requirements 08:12 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 08:13 < jgarzik> I would like to separate things out into multiple tables 08:14 < jgarzik> (as sipa mentioned days ago) 08:15 < sipa> i'm surprised you didn't need to yet 08:15 < sipa> can transactions span multiple tables? 08:16 < jgarzik> yes 08:16 < sipa> in that, you probably want to split it 08:23 < jgarzik> wumpus, One concern with the current implementation is the 'ORDER BY' - a sort - in the CDBIterator class. Once fully sync'd to current bitcoin block height, failing to store in an always-sorted container may create lumpy bitcoind behavior whenever CDBIterator is used... maybe. 08:23 < jgarzik> Needs testing to disprove hypothesis. 08:24 < jgarzik> possibly either the sort is fast enough or smart enough that this is not noticed 08:24 < wumpus> yes, depends on the kind of index 08:25 < wumpus> I suppose the default is a sorted index 08:26 < wumpus> something like a hash index on the key would indeed break the assumption that CDbIterator always returns the results in order 08:27 < jgarzik> well not break - slow down 08:27 < jgarzik> 'ORDER BY' provides the ordering guarantee in case the underlying db does not, in this implementation 08:27 < wumpus> yeah... but an in-database sort of a whole table really isn't pretty 08:27 < jgarzik> nod 08:28 < wumpus> I guess it's best to just use a sorted index for now, unless it proves to be a bottleneck 08:28 < jgarzik> it's a partial sort, starting at the base key 08:28 < jgarzik> not a whole-table sort 08:28 < wumpus> don't we only use iterators over the whole table? 08:28 < jgarzik> not needed 08:29 < jgarzik> code has one pattern: seek(key) then next() next() next() 08:29 < wumpus> then again - without an ordered index, the >= criteria on the key will also cause a full scan 08:29 < jgarzik> yep that is actually an option - assuming results can be unorder - drop 'ORDER BY' and simply scan 08:30 * jgarzik wonders how to tune index types 08:30 < wumpus> yeah, that's the same as the database does internally. I suggest just sticking with a sorted index, unless it turns out to be really a problem, this is unwanted uglyness 08:31 < jcorgan> wumpus, what did you end up with for the list of pragmas 08:32 < jcorgan> WAL isn't making any difference for me 08:33 < wumpus> good question though: do any of the CDBIterator clients require the records to be in a defined order? 08:35 < jgarzik> https://www.sqlite.org/withoutrowid.html 08:35 < wumpus> looks like they don't: their only requirement is that the keys are in a certain range, because the prefix defines what 'table' they are in 08:36 < jgarzik> wumpus, as of now they are within a certain -start- range; end is not excised 08:36 < wumpus> jcorgan: PRAGMA wal_autocheckpoint=0 08:36 < wumpus> jgarzik: the end is too - by breaking out of the iterator as soon as prefix no longer matches 08:36 < jcorgan> got it 08:36 < wumpus> jgarzik: sure, that could be part of the API 08:36 < jgarzik> wumpus, nod - thus sort is required - otherwise iteration ends prematurely 08:37 < jgarzik> it works if you can do ">= start_key" and "< next_prefix" and know what next_prefix is 08:37 < wumpus> my pointi s that the sort would not be required when the records would be binned in a different way, for example in different tables, instead ofusing a prefix to distinguish them 08:37 < wumpus> they don't rely on the fact that keys are sorted 08:37 < jgarzik> wumpus, agreed there - hence I would like to separate things out into multiple tables 08:38 < wumpus> in the case of the UTXO database this doesn't matter that much because it *almost* only contains COINS entries 08:38 < wumpus> for the blockdb on the other hand, it can contain these txindex entries... 08:39 < jgarzik> wumpus, what is your total diff versus 2015_sqlite, WRT pragmas? can you paste that? 08:39 < wumpus> (and a lot of them, the number of block entries is neglible comparison) 08:39 < jgarzik> I want to put that in 2015_sqlite 08:40 < wumpus> + "PRAGMA wal_autocheckpoint=0", 08:40 < wumpus> + "PRAGMA journal_mode=WAL", 08:40 < jcorgan> isn't that supposed to be schema.journal_mode ? 08:41 < jgarzik> global is fine 08:41 < wumpus> I prefer setting the global option 08:41 < jcorgan> ok 08:41 < wumpus> (it's possible to set it per schema) 08:41 < wumpus> (but do we even define those?) 08:42 < jgarzik> no need 08:42 < wumpus> using schema.journal mode *literally* fails, that was my first try too :) 08:42 < jgarzik> wumpus, well, of course it will fail, first time the table does not exist 08:43 < jgarzik> stmts can be reorders but ... setting the global is best 08:43 < wumpus> but setting the global one works, so yeah... 08:43 < jgarzik> reordered 08:44 < jgarzik> OK, pushed out WAL & improved errors to 2015_sqlite 08:44 < jgarzik> going to task switch, poke me if there are other updates for the branch 08:44 < jcorgan> i'll time the total reindex with the new stuff 08:46 < jcorgan> huge difference with the pragmas 08:49 < jgarzik> Thanks. If someone is so motivated, timing with different page sizes (1024 4096, 8192) can be useful. Page size goes all the way up to 64k. 08:50 < wumpus> wouldn't the optimal page size depend on the hardware as well? would be good to do some benchmarks, for example do a full reindex with various sets of parameters, but I don't have a system I can use for controlled tests without background noise 08:51 < wumpus> -stopafterblockimport is a great option for timed, batched reindexes, though 08:53 < jcorgan> i suspect it would be very filesystem and storage configuration dependent 08:56 < wumpus> btw it could be that we need to call sqlite3_wal_checkpoint_v2 at some point (eg, when flushing) with autocheckpointing disabled 08:57 -!- danielsocials [~quassel@222.130.179.179] has quit [Ping timeout: 256 seconds] 08:58 < wumpus> "Writers sync the WAL on every transaction commit if PRAGMA synchronous is set to FULL but omit this sync if PRAGMA synchronous is set to NORMAL." or maybe not. Pragma synchronous is FULL by default. I'm not sure how the checkpoints come into it. 08:58 < wumpus> syncing on transaction commit sounds sane 08:59 < jcorgan> yes 09:04 -!- CodeShark [~CodeShark@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev 09:11 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has joined #bitcoin-core-dev 09:11 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has quit [Changing host] 09:11 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev 09:11 < wumpus> after running a while, bitcoin-shutoff is taking a long time in sqlite3_close - possibly due to the lack of checkpoints 09:14 < wumpus> ... -rw------- 1 39G Oct 24 18:06 db-wal probably 09:16 < wumpus> finished now 09:16 < sipa> ugh 09:17 < jgarzik> "Avoiding Excessively Large WAL Files" is a big section in https://www.sqlite.org/wal.html :) 09:18 < jgarzik> RE optimal page size - ideally it is an "I/O unit" which is filesystem block size - but large dbs might gain from clustering. Also, modern filesystems are kernel-page-cache based, and so an I/O unit in practice is often 4096 or 8192. 09:19 < wumpus> jgarzik: yes - probably setting the page size to anything less than 4096 is not going to be useful 09:20 < wumpus> (at least not on linux) 09:23 < wumpus> adding "rrc = sqlite3_wal_checkpoint_v2(psql, NULL, SQLITE_CHECKPOINT_PASSIVE, NULL, NULL);" at the end of WriteBatch indeed stops the wal file from growing... but, makes the flush slow (~30 seconds) again :/ 09:25 * jgarzik reverted 2015_sqlite back to default autocheckpointing behavior... plenty of room for further research & experiments 09:25 < jgarzik> I think there is a background WAL checkpoint mode 09:25 < wumpus> so I'm not sure when to do these checkpoints 09:25 < wumpus> the default autocheckpointing does a checkpoint every 1000 statements or so, that is probably even worse 09:26 < jgarzik> nod - it's a stable base for further research, not an endpoint 09:26 < jgarzik> the main goal is to not-wait for checkpoint, not not-checkpoint 09:26 < wumpus> yes but we want checkpoints only to happen on commit, not inbetween 09:27 < wumpus> I think the default does so, not sure though... didn't expect manual calls to sqlite3_wal_checkpoint_v2 to be so slow even when done once per transaction 09:28 < wumpus> doing it in the background would be nice 09:28 < wumpus> but then "Checkpointing needs to lock the entire database, so all other readers and writes would have to be blocked. (A passive checkpoint just aborts.)" 09:30 < wumpus> so I'm not sure how much doing the checkpointing in a thread would help in practice. If you do PASSIVE checkpoints they'll never happen at all, and the other modes do waiting in one way or another 09:30 < jcorgan> whouda thunk that getting database operations right would be as hard as getting crypto right? :-) 09:30 < wumpus> well I'm not sure it's as hard as getting crypto right :) but yes it's not trivial 09:31 < wumpus> but seeing all of this it seems leveldb isn't so bad at all 09:32 < jgarzik> switching major database system types, one expects some analysis and bumps in understanding the new system 09:32 < sipa> the only way the wal is cleared is with a checkpoint? 09:32 < wumpus> yes, and this is only an experiment 09:33 < wumpus> sipa: yes, 'checkpoint' seems to be the operation 'incorporate the WAL into the databse' 09:33 < sipa> and that operation needs a full exclusive lock on the database? 09:33 < wumpus> yep 09:33 < sipa> that's unusable 09:34 < jgarzik> The underlying system calls are range locks, so I wonder 09:34 < jgarzik> non-WAL updates the db with range locks, I'm pretty sure 09:34 < sipa> range locks are useless on a db ordered by random indexes 09:35 < wumpus> yes there's always the normal journal, though non-WAL does an fsync per statement, it seems, its performance was really bad here (and all the fsyncs were making the rest of the system slow, too) 09:36 < jgarzik> not necessarily - modern schemes write index/data updates to new pages, allowing older pages to be read in parallel (presenting older, committed view of the data) 09:36 < jgarzik> so you can be writing new while reading old 09:37 < jgarzik> update-in-place is avoided these days, as it does not produce crash-consistent behavior 09:37 < sipa> yes, sure 09:37 < sipa> but you want the background-written log to be incorporated into the real db without locking the whole thing down 09:37 < sipa> leveldb does that 09:38 < jgarzik> nod - and that's perfectly doable here with range locks - you update the new index, then a quick "flick of a switch" jumps from old consistent state to new consistent state, the latter of which was written in parallel in the background 09:39 < sipa> every set of wal will touch database entries all over the place 09:40 < sipa> every non-trivial set.of log entries will need to lock the whole db 09:40 < jgarzik> same is true for every batch, every database system 09:40 < jgarzik> again not true - read the above - you don't need to lock the whole db in theory - I can't speak for sqlite but in database circles the solution is well known here. 09:41 < sipa> oh of.course 09:41 < sipa> i'm just observing sqlite's behaviour 09:41 < sipa> leveldb solves it by having different levels :) 09:41 < sipa> and no guarantee in which level particular data is to be found 09:41 < sipa> socchanges can "ripple up" 09:42 < wumpus> just read https://www.sqlite.org/c3ref/wal_checkpoint_v2.html - from what I understand from it, checkpointing requires an exclusive lock and needs to wait for all readers and writers to finish. If there is a way to work around that it'd be nice, but theory isn't of much help here :) 09:43 < jgarzik> wumpus, that's tuned by wal_checkpoint=[passive/full/restart/truncate] a bit 09:44 < wumpus> tha's mentioned in the link I gave, yes. It looks like that determines who gets to wait (or cancel) for whom, not whether the operation requires an exclusive lock 09:45 < sipa> does issuing a passive checkpoint that gets interrupted make progress? 09:45 < wumpus> it can 09:56 < wumpus> doing e.g. SQLITE_CHECKPOINT_FULL or _TRUNCATE from a thread would make sense I suppose - it blocks all writers and some readers: apparently only those readers not reading from the most recent database snapshot 09:56 < wumpus> at least it wait for all readers to non-recent database snapshot to complete 09:58 < wumpus> (which makes sense as the old version is effectively discarded) 09:59 < wumpus> blocking writers is bad during initial sync, but not so much during steady-state where not much updates happen 10:16 < jcorgan> fyi, this reindex is with txindex=1 10:16 < sipa> that likely hurts performance 10:17 < sipa> txindex writes are not batched 10:17 < sipa> all other database writes are 10:18 < jcorgan> yeah, it's at height 180K and has slowed down dramatically 10:19 < jgarzik> yeah txindex=1 will slow things down & is not representative 10:19 < sipa> i consider txindex something you need for debugging/diagnostics, not for production use 10:19 < wumpus> agreed 10:19 < jcorgan> let me redo it then 10:20 < sipa> that said, it shouldn't gratuitously kill performance if there is an easy way around it 10:21 < sipa> i also doubt that anything you notice at 180k is due to that 10:22 < jcorgan> i don't know that it started at 180k, just that was where it was when i check in on it 10:23 < wumpus> don't you mean 280k? 10:23 < wumpus> (that's where it is here, and I started about the same time) 10:23 < jcorgan> no, 180k 10:23 < wumpus> ok 10:24 < wumpus> unbatched? hmm, doing a sqlite transaction per bitcoin transaction is certainly going to kill sqlite performance 10:25 < jcorgan> restarted fresh with txindex=0, no difference in speed at the start, but we'll see how it slows down 10:25 < wumpus> there's almost no transactions at the start :) 10:25 < jcorgan> right :) 10:26 < jcorgan> it's about 200 blocks/sec 10:26 < CodeShark> I don't even count the first 100k blocks :p 10:30 < CodeShark> so are we benchmarking the SQLite stuff? 10:30 < jcorgan> "benchmarking" would be generous 10:31 < jcorgan> more like, wow, it actually works 10:31 < wumpus> CodeShark: if you want: jgarzik repository, 2015_sqlite branch 10:31 < CodeShark> jcorgan: why wouldn't it work? :) 10:32 < wumpus> it does actually work quite well :) although no one has tested yet if it is more resilient to crashes/poweroffs than leveldb, esp. on windows 10:32 < jcorgan> um, with txindex=0, it just hit 120k in the last 10 minutes 10:32 < CodeShark> sqlite is a pretty solid piece of software (albeit I've never tried using it for something that really taxes it) 10:32 < sipa> jcorgan: that sounds very slow 10:33 < wumpus> this is with autocheckpointing reenabled? 10:33 < jcorgan> i don't think so 10:34 < wumpus> well it won't get faster than that 10:34 < jcorgan> wal, autocheckpoint=0 10:35 < jcorgan> it just hit 180k 10:35 < wumpus> doesn't sound too bad to me 10:35 < jcorgan> still at about 160 block/sec 10:39 < jgarzik> Ideally you want a database system that ensures consistency, permits uninterrupted reads and writes, and optimizes in the background (i.e. "optimize" means move journalled data to final db position, updates indices, etc.) 10:39 < jgarzik> app shouldn't have to checkpoint 10:40 < sipa> yes 10:40 < jcorgan> wumpus: should i have made a further change in the pragmas? 10:41 < jcorgan> also, what was that stop after import option, and can it be set in the .conf file? 10:42 < jgarzik> reads locklessly read consistent data always, and writes do not stomp on that 10:43 < wumpus> jgarzik: they do in sqlite, the checkpoint is just administrative, it doesn't affect what readers read 10:45 < wumpus> at the time checkpoint is called, new readers will already be reading the new state, by the time it can start ,old readers will be finished reading the old state, by definition 10:47 < wumpus> jcorgan: well with autocheckpoint=0 it effectively appends all changes to the db-wal file instead of incorporating them 10:47 < wumpus> this is a matter of where the data is stored, it doesn't affect how clients perceive the database 10:48 < sipa> but reading data from the wal file instead of the actual db must be slower? 10:48 < jgarzik> there is impact if some operations are blocked by other operations 10:48 < wumpus> could be 10:49 < wumpus> there can certainly be performance impact, just not functionality/consistency impact, that's all I'm saying... 10:49 < jgarzik> nod 10:49 < jcorgan> ok, restarted with txindex=0, wal, and removed the autocheckpoint=0 10:50 < wumpus> db-wal for chainstateshould stay at about 440M then 10:50 < wumpus> (1-transaction-sized) 10:51 < wumpus> hmm too big for one transaction.. no I don't know :) 10:53 < wumpus> at least it stays relatively constant around 440mb and doesn't become GB's big anymore, to really make it truncate one'd probably need the SQLITE_CHECKPOINT_TRUNCATE 10:58 < jgarzik> jcorgan, current 2015_sqlite uses autocheckpoint=10000 10:58 < jgarzik> not that I would suggest re-starting yet again :) 10:59 < jcorgan> i have it scripted :) 11:19 < jcorgan> hmm 11:19 < jcorgan> -stopafterblock import shut it down at 183842 11:21 < btcdrak> looks like i am missing the sqlite party 11:21 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev 11:26 -!- maaku [~quassel@botbot.xen.prgmr.com] has joined #bitcoin-core-dev 11:26 -!- maaku is now known as Guest6602 11:27 -!- Guest6602 is now known as maaku 11:29 < CodeShark> it gets really slow once it passes 230k 11:44 -!- maaku [~quassel@botbot.xen.prgmr.com] has left #bitcoin-core-dev ["http://quassel-irc.org - Chat comfortably. Anywhere."] 11:58 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.] 11:58 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 11:59 < jcorgan> something strange is happening 12:00 < jcorgan> the node stops reindexing at 183k, then starts up and asks connections for blocks at this point 12:10 -!- Thireus1 [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 12:11 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has joined #bitcoin-core-dev 12:12 < jcorgan> ok, that's weird 12:12 < jcorgan> somehow i had restored a snapshot of bitcoin dir that only had 183k blocks in it 12:14 < jcorgan> i cloned a new snapshot of the original and it is working fine with that 12:14 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 250 seconds] 12:14 -!- Thireus1 [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection] 12:14 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 12:16 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Client Quit] 12:16 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 12:18 -!- Thireus1 [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 12:21 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 240 seconds] 12:43 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Quit: Leaving] 12:46 < btcdrak> I rebased #6312 and #6564 (BIP68+CSV) into one branch for the people who asked to review both PRs in one changeset https://github.com/bitcoin/bitcoin/compare/master...btcdrak:sequenceandcsv 12:57 -!- malte [~malte@2a00:d0c0:200:0:b9:1a:9c2c:1] has quit [Ping timeout: 250 seconds] 12:57 -!- malte [~malte@2a00:d0c0:200:0:b9:1a:9c2c:1] has joined #bitcoin-core-dev 13:00 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.] 13:10 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 13:10 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has joined #bitcoin-core-dev 13:14 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has quit [Client Quit] 13:15 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has joined #bitcoin-core-dev 13:26 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 13:34 -!- fkhan [weechat@gateway/vpn/mullvad/x-rymbotvculqltgnb] has quit [Ping timeout: 260 seconds] 13:46 -!- fkhan [weechat@gateway/vpn/mullvad/x-yjxxwavjvubkbjnz] has joined #bitcoin-core-dev 13:52 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev 14:02 -!- Thireus1 [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.] 14:02 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:07 -!- Thireus1 [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:07 -!- Thireus1 [~Thireus@icy.thireus.fr] has quit [Client Quit] 14:07 -!- Thireus1 [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:08 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 246 seconds] 14:11 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:12 -!- Thireus1 [~Thireus@icy.thireus.fr] has quit [Ping timeout: 246 seconds] 14:12 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 14:12 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Client Quit] 14:12 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:14 -!- Thireus1 [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:17 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 250 seconds] 14:24 -!- Thireus1 [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.] 14:25 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:28 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Client Quit] 14:28 -!- Thireus1 [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:30 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 14:30 -!- Thireus1 [~Thireus@icy.thireus.fr] has quit [Client Quit] 14:44 -!- rcutmore [~rcutmore@172.77.36.248] has joined #bitcoin-core-dev 14:59 -!- gmaxwell is now known as gmaxweIl 14:59 -!- gmaxweIl is now known as gmaxwell 15:05 -!- nkuttler [~nkuttler@unaffiliated/nkuttler] has joined #bitcoin-core-dev 15:10 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev 15:10 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has joined #bitcoin-core-dev 15:16 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 15:46 -!- ParadoxSpiral_ [~ParadoxSp@p508B9F77.dip0.t-ipconnect.de] has quit [Quit: cya] 15:46 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev 15:57 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has quit [Quit: Later.] 15:59 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has joined #bitcoin-core-dev 16:07 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 16:57 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has quit [Read error: Connection reset by peer] 16:58 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has joined #bitcoin-core-dev 17:11 -!- gmaxwell is now known as gmaxweii 17:11 -!- gmaxweii is now known as gmaxweiI 17:11 -!- gmaxweiI is now known as gmaxwell_ 17:11 -!- gmaxwell_ is now known as gmaxwelI 17:12 -!- gmaxwelI is now known as gmaxwell 17:15 -!- danielsocials [~quassel@222.130.179.179] has joined #bitcoin-core-dev 17:19 -!- belcher is now known as beIcher 17:19 -!- beIcher is now known as belcher 17:31 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has quit [Ping timeout: 246 seconds] 17:54 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-iexcitcqxjxehrvv] has quit [Quit: Connection closed for inactivity] 18:09 < GitHub74> [bitcoin] MarcoFalke opened pull request #6887: [qt] Update coin control and smartfee labels (master...MarcoFalke-2015-qtMaxMin_Fee_and_Max_Fee) https://github.com/bitcoin/bitcoin/pull/6887 18:14 -!- danielsocials [~quassel@222.130.179.179] has quit [Ping timeout: 265 seconds] 18:16 -!- alpalp [6836eb1c@gateway/web/cgi-irc/kiwiirc.com/ip.104.54.235.28] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 18:21 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has joined #bitcoin-core-dev 18:37 -!- danielsocials [~quassel@222.130.179.179] has joined #bitcoin-core-dev 18:58 -!- danielsocials [~quassel@222.130.179.179] has quit [Ping timeout: 255 seconds] 19:07 -!- danielsocials [~quassel@222.130.179.179] has joined #bitcoin-core-dev 19:26 -!- vbuilderv [~JoeLiu@110.188.198.64] has joined #bitcoin-core-dev 19:30 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Ping timeout: 250 seconds] 19:36 -!- danielsocials [~quassel@222.130.179.179] has quit [Ping timeout: 272 seconds] 20:03 -!- belcher [~user@unaffiliated/belcher] has quit [Read error: Connection reset by peer] 20:14 -!- danielsocials [~quassel@45.32.248.113] has joined #bitcoin-core-dev 20:17 -!- danielsocials [~quassel@45.32.248.113] has quit [Remote host closed the connection] 20:32 -!- vbuilderv [~JoeLiu@110.188.198.64] has quit [Quit: Leaving.] 21:45 -!- helo [~helo@unaffiliated/helo] has quit [Ping timeout: 260 seconds] 21:47 -!- helo [~helo@unaffiliated/helo] has joined #bitcoin-core-dev 22:16 -!- danielsocials [~quassel@45.32.248.113] has joined #bitcoin-core-dev 22:20 -!- danielsocials [~quassel@45.32.248.113] has quit [Remote host closed the connection] 22:50 < CodeShark> I'm experiencing a very strange travis issue - everything works fine with one commit, then I just add this https://github.com/CodeShark/bitcoin/commit/6f42f34f03834b67cc09b53a3809bee9d2239b8e and suddenly I get the seemingly unrelated error: test/alert_tests.cpp(223): error in "PartitionAlert": check strMiscWarning.empty() failed - https://s3.amazonaws.com/archive.travis-ci.org/jobs 22:50 < CodeShark> /87276294/log.txt 22:51 < CodeShark> the previous commit has no issues: https://github.com/bitcoin/bitcoin/pull/6816/commits 22:53 < CodeShark> I'm completely stumped on this one 23:10 < phantomcircuit> jgarzik, you can do INSERT INTO table (column, column) VALUES (1,2),(3,4); 23:10 < phantomcircuit> batch the batches 23:10 < sipa> i doubt that makes any difference 23:14 < phantomcircuit> also you need to assert the returned data type 23:15 < phantomcircuit> sqlite doesn't strictly enforce the data types... 23:17 < phantomcircuit> I think there is a background WAL checkpoint mode 23:17 < phantomcircuit> there isn't you have to do it yourself 23:21 < phantomcircuit> sipa, it can make a difference if the batches are very large 23:23 < phantomcircuit> you get the best performance with a wal and a background thread that's doing passive checkpoints in a loop 23:23 < phantomcircuit> iirc there's an api call to get stats on the journal which can tell you if a SQLITE_CHECKPOINT_FULL would be fast 23:24 < phantomcircuit> it'll still be slow probably though 23:25 < phantomcircuit> right it's the two output values from sqlite3_wal_checkpoint_v2 23:25 < phantomcircuit> pnLog tells you how many frames there are and pnCkpt tells you how many have been checkpointed 23:25 < phantomcircuit> nLog - nCkpt == frames left to checkpoint 23:34 -!- crescendo [~mozart@unaffiliated/crescendo] has quit [Ping timeout: 240 seconds] 23:35 -!- crescendo [~mozart@unaffiliated/crescendo] has joined #bitcoin-core-dev 23:39 < CodeShark> hmmm - this won't work for versionbits at all: https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L2034 23:42 < sipa> bip9 specifies a replacement for the alert logic 23:45 < gmaxwell> heh. 25% of the design of versionbits is specificlly there to facilitate smarter warning behavior. 23:52 < CodeShark> the current spec says we should only warn if a lock-in is detected for an unknown bit - so if the bit is set in 94% of blocks for an interval no warning? 23:53 < CodeShark> I was thinking to warn if any unknown bit is detected...but that would make it easy for miners to trip off a bunch of clients with a single block 23:55 < CodeShark> furthermore, it presumes we can detect a lock-in for a fork we haven't even defined...so this would force the threshold to actually be a chain parameter (and would potentially complicate matters if we ever want to modify thresholds) 23:56 < CodeShark> I'd say we should warn if at least half of the blocks within an interval show an unknown bit 23:57 < gmaxwell> CodeShark: the document talks about this. :( 23:58 < CodeShark> yes, I'm referring to the document 23:59 < gmaxwell> The thresholds are specified but they are not consensus rules. If one uses another threshold, one can do so-- but one has to cope with whatever impact that has on warning behavior. 23:59 < gmaxwell> https://github.com/bitcoin/bips/blob/master/bip-0009.mediawiki#Support_for_future_changes