--- Day changed Thu Nov 05 2015 00:08 < jonasschnelli> wumpus: what do you think by adding https://github.com/bitcoin/bitcoin/pull/6948 to 6917? It would at least allow me to easily generate a build over gitian. 00:17 < GitHub187> [bitcoin] laanwj closed pull request #6919: Backport chainstate obfuscation to 0.11 (0.11...2015_10_backport_chainstate_obfuscation) https://github.com/bitcoin/bitcoin/pull/6919 00:23 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 00:36 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has quit [Ping timeout: 250 seconds] 00:43 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Remote host closed the connection] 01:06 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 01:29 -!- deepcore [~deepcore@2a01:79d:469e:ed94:8e70:5aff:fe5c:ae78] has joined #bitcoin-core-dev 01:50 < GitHub127> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/193f7b553e0a...79456524f8f5 01:50 < GitHub127> bitcoin/master fb9857b Pieter Wuille: Squashed 'src/leveldb/' changes from 7d41e6f..20ca81f... 01:50 < GitHub127> bitcoin/master f0343e9 Pieter Wuille: Update LevelDB 01:50 < GitHub127> bitcoin/master 7945652 Wladimir J. van der Laan: Merge pull request #6944... 01:50 < GitHub14> [bitcoin] laanwj closed pull request #6944: Update LevelDB tree to include #6917 (master...leveldbfix0.12) https://github.com/bitcoin/bitcoin/pull/6944 01:50 < wumpus> jonasschnelli: this works, too 01:52 < GitHub57> [bitcoin] laanwj pushed 3 new commits to 0.11: https://github.com/bitcoin/bitcoin/compare/dfe55bdc32b5...2c8248552492 01:52 < GitHub87> [bitcoin] laanwj closed pull request #6945: Update LevelDB tree to include #6917 (0.11) (0.11...leveldbfix0.11) https://github.com/bitcoin/bitcoin/pull/6945 01:52 < GitHub57> bitcoin/0.11 0af5b8e Pieter Wuille: Squashed 'src/leveldb/' changes from 7d41e6f..20ca81f... 01:52 < GitHub57> bitcoin/0.11 70de437 Pieter Wuille: Update LevelDB 01:52 < GitHub57> bitcoin/0.11 2c82485 Wladimir J. van der Laan: Merge pull request #6945... 01:52 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has joined #bitcoin-core-dev 01:53 < GitHub53> [bitcoin] laanwj pushed 3 new commits to 0.10: https://github.com/bitcoin/bitcoin/compare/72a0adfb3c36...cbc4e3bd37da 01:53 < GitHub53> bitcoin/0.10 5216f3c Pieter Wuille: Squashed 'src/leveldb/' changes from 7d41e6f..20ca81f... 01:53 < GitHub53> bitcoin/0.10 94b67e5 Pieter Wuille: Update LevelDB 01:53 < GitHub53> bitcoin/0.10 cbc4e3b Wladimir J. van der Laan: Merge pull request #6946... 01:53 < GitHub168> [bitcoin] laanwj closed pull request #6946: Update LevelDB tree to include #6917 (0.10) (0.10...leveldbfix0.10) https://github.com/bitcoin/bitcoin/pull/6946 02:06 < jonasschnelli> wumpus: okay. So building https://github.com/bitcoin/bitcoin/pull/6948 on top of master should include #6917 i guess 02:07 < wumpus> jonasschnelli: yep 02:07 * jonasschnelli has kicked his gitian builder 02:08 < wumpus> yes that reminds me that gitian doesn't do any merging/rebasing like the travis build does, so having #6917 in master wouldn't actually help by default 02:09 -!- rubensayshi [~ruben@91.206.81.13] has joined #bitcoin-core-dev 02:17 -!- deepcore [~deepcore@2a01:79d:469e:ed94:8e70:5aff:fe5c:ae78] has quit [Ping timeout: 264 seconds] 02:19 < jonasschnelli> wumpus : IIRC my script does add PRs on top of master before building the custom git state: https://bitcoin.jonasschnelli.ch/pulls/6917/ 02:20 < wumpus> oh, cool! 02:24 -!- deepcore [~deepcore@2a01:79d:469e:ed94:8e70:5aff:fe5c:ae78] has joined #bitcoin-core-dev 03:04 < btcdrak> wumpus: ping mempool-MPL revert undo and 0.11 backport #6934, #6884 03:11 < GitHub30> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/79456524f8f5...3694b74fa9c8 03:11 < GitHub30> bitcoin/master abd8b76 MarcoFalke: [qt] Properly display required fee instead of minTxFee 03:11 < GitHub30> bitcoin/master 53238ff MarcoFalke: Clarify what minrelaytxfee does 03:11 < GitHub30> bitcoin/master 3694b74 Wladimir J. van der Laan: Merge pull request #6887... 03:11 < GitHub59> [bitcoin] laanwj closed pull request #6887: [qt] Update coin control and smartfee labels (master...MarcoFalke-2015-qtMaxMin_Fee_and_Max_Fee) https://github.com/bitcoin/bitcoin/pull/6887 03:12 < GitHub35> [bitcoin] laanwj closed pull request #6726: AcceptToMemoryPool: Don't fee-check wallet-created transactions (master...MarcoFalke-2015-rejectWtxFix) https://github.com/bitcoin/bitcoin/pull/6726 03:13 < GitHub47> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/3694b74fa9c8...3038eb63e8a6 03:13 < GitHub47> bitcoin/master e4e5334 Gregory Maxwell: Restore MedianTimePast for locktime.... 03:13 < GitHub47> bitcoin/master d1c3762 Gregory Maxwell: Revert "Revert "Enable policy enforcing GetMedianTimePast as the end point of lock-time constraints""... 03:13 < GitHub47> bitcoin/master 3038eb6 Wladimir J. van der Laan: Merge pull request #6934... 03:13 < GitHub106> [bitcoin] laanwj closed pull request #6934: Restores mempool only BIP113 enforcement (master...undo_undo) https://github.com/bitcoin/bitcoin/pull/6934 03:18 < GitHub41> [bitcoin] laanwj pushed 3 new commits to 0.11: https://github.com/bitcoin/bitcoin/compare/2c8248552492...df616ae43ed4 03:18 < GitHub115> [bitcoin] laanwj closed pull request #6884: Backport #6566, median-past locktime, rebased against 0.11 (0.11...mpl-0.11) https://github.com/bitcoin/bitcoin/pull/6884 03:18 < GitHub41> bitcoin/0.11 a1d3c6f Mark Friedenbach: Add rules--presently disabled--for using GetMedianTimePast as endpoint for lock-time calculations... 03:18 < GitHub41> bitcoin/0.11 f720c5f Mark Friedenbach: Enable policy enforcing GetMedianTimePast as the end point of lock-time constraints... 03:18 < GitHub41> bitcoin/0.11 df616ae Wladimir J. van der Laan: Merge pull request #6884... 04:11 < jtimon> gmaxwell: re transaction cost formula: when you use it only as policy, it has nothing to do with the blocksize limit, when you use it to calculate the size of blocks it does (that's whatI meant by "we can make it consensus") 05:20 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has joined #bitcoin-core-dev 05:21 -!- fanquake [~Adium@unaffiliated/fanquake] has joined #bitcoin-core-dev 05:23 < fanquake> Has anyone been testing core with Boost 1.59.0 ? 05:25 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has quit [Ping timeout: 264 seconds] 05:25 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev 05:27 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has joined #bitcoin-core-dev 05:32 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has quit [Ping timeout: 250 seconds] 05:33 < jonasschnelli> wumpus: still facing "Corruption: error in middle of record" with #6948 05:34 < wumpus> fuck fuck fuck 05:34 < wumpus> fanquake: yes, works fine here 05:34 * jonasschnelli is still on boost 1.58 05:35 < jonasschnelli> wumpus: also did shut down / power off the host (mac) and there where no issues with the database. 05:35 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has joined #bitcoin-core-dev 05:36 < wumpus> well I never managed to get an error in the middle of record with #6948, so at least it should be more rare now 05:36 < jonasschnelli> Steps i did (maybe i did something wrong): my snapshot has a synced chain up to ~350000 form PR https://github.com/bitcoin/bitcoin/pull/6917 05:36 < jonasschnelli> then i stopped bitcoin-qt and started bitcoin-qt from https://github.com/bitcoin/bitcoin/pull/6948 05:36 < jonasschnelli> did a power off hard shutdown 05:36 < jonasschnelli> restarted win8.1 ... restarted bitcoin-qt from 6948 05:36 < jonasschnelli> -> resulted in "Corruption: error in middle of record" 05:37 < wumpus> but if that doesn't fix it I don't know, I really don't know anymore 05:37 < jonasschnelli> switch away from leveldb 05:38 < wumpus> maybe FlushFileBuffers on windows doesn't really flush the buffers 05:38 < wumpus> no, I don't want to hear that 05:38 < jonasschnelli> I'll do the test again ... maybe i missclicked somewhere. 05:38 < wumpus> well, unless, have you tried this with another database at all? 05:39 < jonasschnelli> sqlite3 was to slow i heard... i'd though about trying it with the unstable sqlite4 branch 05:39 < wumpus> it would be somewhat interesting to try this with jgarzik's sqlite patch. At least to have a measurement point. 05:39 < jonasschnelli> jgarzik uses seqlite3 which is certenly much slower then leveldb... sqlite4 sounds promising... and unstable. 05:40 < wumpus> for this experiment I don't care about slowness - I wonder if *anything * can handle this kind of corruption succesfully 05:40 < jonasschnelli> when i restore my snapshot based on a ~synced chain with #6917, bitcoin-core crashed,... but can be started again with 100% success. Can that be a problem? I assume no? 05:41 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has quit [Ping timeout: 272 seconds] 05:42 < fanquake> jonasschnelli Do you have VM snapshots available for download? I can contribute some more time to testing this, having a VM to boostrap from would be handy. 05:42 -!- fanquake [~Adium@unaffiliated/fanquake] has left #bitcoin-core-dev [] 05:43 -!- fanquake [~Adium@unaffiliated/fanquake] has joined #bitcoin-core-dev 05:43 < jonasschnelli> fanquake: I have plenty of VMs (multiple harddrives full of vms), just tell me what you need. I can even get some of you access to VMs if someone needs to track down an issue. 05:43 < jonasschnelli> Maybe you'd like to start with the Win8.1 VM... you need VMWare Fusion 8 (or higher) 05:44 < fanquake> I'd take copies of the ones your using to test #6948 05:44 < wumpus> I suppose what leveldb should do when it encounters corruption in the middle of a record (probably a run of 0000000 like debug.log) is to simply discard that batch and anything after it 05:44 < fanquake> I'm actually setup somewhere with a decent internet connection atm, so downloading won't be so hard. 05:44 < fanquake> Can't run in virtualbox? 05:44 < jonasschnelli> nope. 05:45 < fanquake> Ah ok, I'll spin up something on VB then. 05:45 < wumpus> be careful to actually test #6948 on top of master, it's currently rebased to a pre-#6917 branch so as-is it won't test the robust version 05:46 < jonasschnelli> Yes. I'm using https://bitcoin.jonasschnelli.ch/pulls/6948/... has 22e780737db57bcb18b3824eb8158e19a4775cb6 and fb9857bfd68c13b52e14bc28dd981bc12501806a 05:46 < fanquake> wumpus I'll also get back to you about the gitian switch to trusty 05:46 * jonasschnelli is starting bitcoin-qt after a power off hard shutdown... 05:47 < jonasschnelli> "Error opening block database" 05:47 < fanquake> What resources are you giving your vms? Just curious. 05:47 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has joined #bitcoin-core-dev 05:48 < jonasschnelli> 1GB Ram, 1 Core... not that much. But i have bought the wrong MacMini (4GB Ram, Only 1.4Ghz Core i5). 05:48 < jonasschnelli> It has soldered RAM... tzz,.. apple! 05:48 < fanquake> heh 05:48 < fanquake> I'm setting up a VM now, so will start testing with your pull-tester build. 05:51 < jonasschnelli> i don't have the 0x00 in the debug log... it has a proper end before it starts with the app run that shows up the corruption 05:53 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has quit [Ping timeout: 265 seconds] 05:56 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has joined #bitcoin-core-dev 05:57 < wumpus> fanquake: thanks 05:58 < jonasschnelli> i now do a local depends windows cross compile and try to get some additional debug info... 06:00 < jonasschnelli> wumpus: did you once try the FILE_FLAG_NO_BUFFERING flag? 06:02 < wumpus> nope. 06:07 -!- Guest7284 [~pigeons@94.242.209.214] has quit [Ping timeout: 268 seconds] 06:08 -!- pigeons [~pigeons@94.242.209.214] has joined #bitcoin-core-dev 06:08 -!- pigeons is now known as Guest57961 06:11 -!- qpm [~qpm@unaffiliated/midnightmagic/bot/qpm] has quit [Remote host closed the connection] 06:12 < wumpus> it's worth trying, also you could try putting back the FlushFileBuffer in Flush () (eg https://github.com/bitcoin/bitcoin/pull/6917#issuecomment-152717077 ) 06:16 < jonasschnelli> okay... will try 06:16 < jonasschnelli> (first need to build the depends... *sleep*) 06:24 < jonasschnelli> hmm... can't anymore build the dependencies with HOST i686-w64-mingw32... 06:24 < jonasschnelli> qwindowsdialoghelpers.cpp:1849:92: error: invalid conversion from ‘qt_LpItemIdList {aka _ITEMIDLIST*}’ to ‘PCIDLIST_ABSOLUTE 06:31 < jgarzik> jonasschnelli, RE databases and comparison... I should have a fast replacement ready in a few weeks, https://github.com/jgarzik/pgdb2 06:31 < jgarzik> jonasschnelli, sqlite is probably too slow for useful benchmarking 06:31 < jonasschnelli> jgarzik: hah. Let me check out the code... 06:32 < jonasschnelli> jgarzik: what about sqlite4? It's says on the architecture website ("It is very fast, faster than LevelDB...",) 06:33 < jonasschnelli> And riding on one of the best maintained database layers would be nice... but right,.. it might be the wrong type of database 06:33 < jgarzik> jonasschnelli, https://sqlite.org/src4/doc/trunk/www/design.wiki definitely looks like they are catching up to modern design 06:34 < jonasschnelli> I have also though about your "pgdb2"-like-approach, ... but it seems to be a hard way. 06:34 < jgarzik> "The default built-in storage engine is a log-structured merge database. It is very fast, faster than LevelDB, supports nested transactions, and stores all content in a single disk file. " 06:35 * jgarzik knows what log-structured is. What is a merge database, I wonder 06:36 < jonasschnelli> jgarzik: would it be complicated to modificate your sqlite3 branch to use sqlite4? (over a subtree or something similar because it's unstable and not available over dep.management?) 06:36 < jgarzik> jonasschnelli, Part of pgdb2 is building an underlying framework that can be used for ordered-map key/value, hash tree key/value, merkle db and more - a bit of a primitive kernel filesystem that provides transactions/COW to the upper layer, which defines the key/value structure. It's the same speed (or faster, really) 06:37 < jgarzik> jonasschnelli, sounds like using sqlite4 is simple - take my branch and add 'env' argument to each function call. 06:38 < jonasschnelli> jgarzik: do you have plans to work towards sqlite4 and report some benchmark numbers? Or should i give it a try? 06:39 < jonasschnelli> but i really like to find a solution to fix the current leveldb corruption issues... 06:39 < jonasschnelli> although, i don't care about windows enough,.... but i care about users complaining about the issue and stopping full nodes. 06:40 < jgarzik> jonasschnelli, feel free to give sqlite4 a try. I consider my sqlite branch at a [temporary?] pause / end state. I task switched over to BIP 1xx and pgdb2. 06:41 < jonasschnelli> Okay. pgdb2 looks nice btw! 06:41 < sipa> jgarzik: merge database... there are several files that can refer to the same range of keys, but if there are too many you merge them into a bigger file 06:42 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Quit: Leaving] 06:42 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev 06:42 < jgarzik> jonasschnelli, go ahead and try with sqlite4 06:42 < jgarzik> jonasschnelli, sqlite3 branch is at a temporary end state, unless further speedups can be found 06:43 < jgarzik> jonasschnelli, I task switched over to BIP 1xx and pgdb2 06:44 -!- warren [~warren@fedora/wombat/warren] has quit [Ping timeout: 250 seconds] 06:44 -!- crescendo [~mozart@unaffiliated/crescendo] has quit [Ping timeout: 250 seconds] 06:45 -!- crescendo [~mozart@unaffiliated/crescendo] has joined #bitcoin-core-dev 06:46 < sipa> jonasschnelli: #6948 is only expected to solve (or improve at least) the undo corruption, not leveldb middle of record stuff 06:46 -!- Luke-Jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood] 06:46 -!- Luke-Jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-dev 06:47 -!- warren [~warren@fedora/wombat/warren] has joined #bitcoin-core-dev 06:47 < jonasschnelli> sipa: Right. I see. I've thought together with #6917 (leveldb) it would fix the whole corruption issues. Because with 6917 i "mostly" got the "bad undo" error... but now i get 100% leveldb middle of records errror. 06:48 < sipa> are you sure you did #6948 together with #6917? 06:48 < sipa> and not only #6948? 06:56 -!- Guyver2 [~CK@guyver2.xs4all.nl] has joined #bitcoin-core-dev 07:04 < GitHub50> [bitcoin] morcos closed pull request #6857: Require nLastBlockFile to be the highest numbered file. (master...nLastBlockFile) https://github.com/bitcoin/bitcoin/pull/6857 07:21 -!- ParadoxSpiral [~ParadoxSp@p508B9536.dip0.t-ipconnect.de] has joined #bitcoin-core-dev 07:43 < jonasschnelli> sipa: Yes. https://bitcoin.jonasschnelli.ch/pulls/6948/ Is the current master with your fixflush branch 07:48 < jonasschnelli> anyone else facing qt5 depends build issues when building with mingw on debian or ubuntu? 08:12 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: Textual IRC Client: www.textualapp.com] 08:12 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev 08:20 -!- fanquake [~Adium@unaffiliated/fanquake] has quit [Quit: Leaving.] 08:33 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.] 08:51 -!- rubensayshi [~ruben@91.206.81.13] has quit [Ping timeout: 240 seconds] 08:52 -!- dhill_ [~dhill@phengo.phobia.ms] has joined #bitcoin-core-dev 08:52 -!- dhill [~dhill@phengo.phobia.ms] has quit [Killed (adams.freenode.net (Nickname regained by services))] 08:52 -!- dhill_ is now known as dhill 08:56 -!- wump [~quassel@pdpc/supporter/professional/wumpus] has joined #bitcoin-core-dev 09:00 -!- zmanian_ [uid113594@gateway/web/irccloud.com/x-ldinrjkhltfwqnoc] has joined #bitcoin-core-dev 09:00 -!- banana_lotus [~BananaLot@54.186.186.141] has joined #bitcoin-core-dev 09:01 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Remote host closed the connection] 09:02 -!- davec is now known as btcdbot 09:03 -!- btcdbot is now known as davec 09:04 -!- davec is now known as btcdbot 09:04 -!- btcdbot is now known as davec 09:04 -!- Netsplit *.net <-> *.split quits: wumpus, [b__b], zmanian, BananaLotus, amiller 09:05 -!- banana_lotus is now known as BananaLotus 09:05 -!- amiller_ [~socrates1@li175-104.members.linode.com] has joined #bitcoin-core-dev 09:05 -!- zooko [~user@c-73-14-172-248.hsd1.co.comcast.net] has joined #bitcoin-core-dev 09:05 -!- wump is now known as wumpus 09:07 -!- Netsplit over, joins: [b__b] 09:11 -!- deepcore [~deepcore@2a01:79d:469e:ed94:8e70:5aff:fe5c:ae78] has quit [Quit: WeeChat 1.3] 09:20 -!- CodeShark [~CodeShark@2600:380:4b43:bae5:fd25:a062:1823:9dda] has joined #bitcoin-core-dev 09:30 -!- zooko [~user@c-73-14-172-248.hsd1.co.comcast.net] has quit [Ping timeout: 264 seconds] 09:38 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has joined #bitcoin-core-dev 09:38 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has quit [Changing host] 09:38 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev 09:46 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 09:59 -!- zooko [~user@c-73-14-172-248.hsd1.co.comcast.net] has joined #bitcoin-core-dev 10:04 -!- molly [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds] 10:05 -!- zooko [~user@c-73-14-172-248.hsd1.co.comcast.net] has quit [Remote host closed the connection] 10:13 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 10:26 < GitHub113> [bitcoin] MarcoFalke opened pull request #6951: [qt] Use maxTxFee instead of 10000000 (master...MarcoFalke-2015-qtMaxFee) https://github.com/bitcoin/bitcoin/pull/6951 10:28 -!- zooko [~user@50.141.118.222] has joined #bitcoin-core-dev 10:29 < gmaxwell> jtimon: I am not suggesting it for using it with blocks. I think it would be bad for that. 10:32 < gmaxwell> wumpus: for all you know there is something wrong with jonasschnelli's VM and it impermisably reorders writes or ignores sync. 10:33 < gmaxwell> wumpus: (I would be completely unsurprised to see VM software which ignored sync to prevent vm guest OS from thrashing the host... and used userspace buffering of writes for similar reasons, and didn't guarentee them flushed when it quit) 10:34 < gmaxwell> wumpus: in any case, there is a clear path forward, get the corrupted data from him and check it out. 10:35 < gmaxwell> wumpus: based on the undo related errors before I believe his node is in the process of a reindex as he's doing these tests? 10:35 < gmaxwell> If it isn't the VM itself that is the difference in your test conditions, then it may be reindexing. 10:40 < wumpus> true, that may be the case 10:41 < wumpus> going to test master+#6948 on real-windows tomorrow, if I can't get corruption, I'll just ignore the VM results for now 10:42 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev 10:47 < jgarzik> some VMs definitely do that 10:55 < jonasschnelli> Okay. This might be the case (although I never expected any corruption while hard power off a VM). 11:00 < GitHub112> [bitcoin] TheBlueMatt closed pull request #6595: Fix removal of timelocked-txn from mempool during reorg (master...lockedmempool) https://github.com/bitcoin/bitcoin/pull/6595 11:05 -!- zooko [~user@50.141.118.222] has quit [Ping timeout: 250 seconds] 11:07 -!- molly [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 11:20 < jtimon> gmaxwell: I know, you are suggesting to use it for acceptToMemoryPool 11:56 < phantomcircuit> i doubt this is an actual bug as the datadir it's reading from has had lots of pretty random patches run against it but maybe ... http://0bin.net/paste/AYFDFisRwwdGNPCB#EhQRzbuLtMfbjQ0SSfv+HZasT4HPN8nBM9i2Jutkufa 11:57 < phantomcircuit> Assertion `hashPrevBlock == view.GetBestBlock()' 12:01 -!- CodeShark_ [~androirc@40.135.239.174] has joined #bitcoin-core-dev 12:10 < GitHub136> [bitcoin] luke-jr opened pull request #6952: Backport bugfixes to 0.10 (2015-10-22 / f2c869a) (master...backport-bugfixes-to-0.10-20151014) https://github.com/bitcoin/bitcoin/pull/6952 12:10 < GitHub117> [bitcoin] luke-jr closed pull request #6952: Backport bugfixes to 0.10 (2015-10-22 / f2c869a) (master...backport-bugfixes-to-0.10-20151014) https://github.com/bitcoin/bitcoin/pull/6952 12:11 < GitHub125> [bitcoin] luke-jr opened pull request #6953: Backport bugfixes to 0.10 (2015-10-22 / f2c869a) (0.10...backport-bugfixes-to-0.10-20151014) https://github.com/bitcoin/bitcoin/pull/6953 12:11 < wumpus> phantomcircuit: probably you're trying to open an obfuscated data directory in an earlier release 12:12 < wumpus> (happens when you start a database with master and try to open it with 0.11.x, for example) 12:12 < phantomcircuit> wumpus, ah yeah that's it 12:17 < GitHub181> [bitcoin] TheBlueMatt closed pull request #5347: Make mapNextTx private within CTxMemPool (master...mapnexttxpriv) https://github.com/bitcoin/bitcoin/pull/5347 12:18 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [] 12:31 < GitHub133> [bitcoin] TheBlueMatt closed pull request #5433: Make mempool-removed tracking much more explicit (master...mempoolconflict) https://github.com/bitcoin/bitcoin/pull/5433 12:32 < GitHub182> [bitcoin] sipa opened pull request #6954: Switch to libsecp256k1-based ECDSA validation (master...secp256k1) https://github.com/bitcoin/bitcoin/pull/6954 12:39 < sdaftuar> sipa: nice! 12:42 < jonasschnelli> sipa: awesome work #6954 12:42 * jonasschnelli started building 6954 and can't wait to compare IBD time... 12:43 < sipa> jonasschnelli: increase -dbcache :) 12:44 < jonasschnelli> what sizes are ideal? Allthough, I don't want to do that on my 1GB VMs... 12:44 < sdaftuar> i think you want at least 3GB ideally 12:46 * jonasschnelli bought the wrong macmini for testing,... soldered 4GB RAM! *grml* 12:46 < sdaftuar> grr. literally as i typed that, my -reindex running with 4GB dbcache just flushed to disk! sigh 12:46 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Ping timeout: 246 seconds] 12:47 < gmaxwell> utxo set has been growing a lot lately, 3gb used to be enough.. a couple months ago. 12:47 < gmaxwell> since most of the blocks are used up by utxo bloating attack txn. :( 13:01 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev 13:40 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: Textual IRC Client: www.textualapp.com] 13:42 < GitHub29> [bitcoin] MarcoFalke opened pull request #6955: [trivial] White space, clang-format and docs (master...MarcoFalke-2015-trivial4) https://github.com/bitcoin/bitcoin/pull/6955 13:50 -!- ParadoxSpiral [~ParadoxSp@p508B9536.dip0.t-ipconnect.de] has quit [Remote host closed the connection] 14:06 < GitHub118> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/3038eb63e8a6...849a7e645323 14:06 < GitHub118> bitcoin/master 22e7807 Pieter Wuille: Always flush block and undo when switching to new file... 14:06 < GitHub118> bitcoin/master 849a7e6 Wladimir J. van der Laan: Merge pull request #6948... 14:09 -!- CodeShark [~CodeShark@2600:380:4b43:bae5:fd25:a062:1823:9dda] has quit [Ping timeout: 240 seconds] 14:09 -!- Luke-Jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 240 seconds] 14:10 -!- Luke-Jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-dev 14:23 -!- PRab [~chatzilla@2601:40a:8000:8f9b:d59b:22a1:99bf:deb1] has quit [Read error: Connection reset by peer] 14:24 -!- PRab [~chatzilla@2601:40a:8000:8f9b:d59b:22a1:99bf:deb1] has joined #bitcoin-core-dev 14:30 -!- [b__b] [~b__b]@ec2-54-85-45-223.compute-1.amazonaws.com] has quit [Remote host closed the connection] 14:30 -!- [b__b] [~b__b]@ec2-54-85-45-223.compute-1.amazonaws.com] has joined #bitcoin-core-dev 14:31 -!- CodeShark [~CodeShark@2600:380:4b43:bae5:fd25:a062:1823:9dda] has joined #bitcoin-core-dev 14:40 < GitHub190> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/849a7e645323...4ee149a6db25 14:40 < GitHub190> bitcoin/master 0af8fe4 MarcoFalke: devtools: Update README.md 14:40 < GitHub190> bitcoin/master e0eeb67 MarcoFalke: [trivial] clang-format: Set AlignAfterOpenBracket: false 14:40 < GitHub190> bitcoin/master e167af2 MarcoFalke: [doc] Remove excessive white space 14:40 < GitHub152> [bitcoin] laanwj closed pull request #6955: [trivial] White space, clang-format and docs (master...MarcoFalke-2015-trivial4) https://github.com/bitcoin/bitcoin/pull/6955 14:40 -!- Guyver2 [~CK@guyver2.xs4all.nl] has quit [Quit: :)] 14:44 -!- MarcoFalke [50edb194@gateway/web/cgi-irc/kiwiirc.com/ip.80.237.177.148] has joined #bitcoin-core-dev 14:47 -!- phantomcircuit [~phantomci@strateman.ninja] has quit [Quit: quit] 14:52 -!- CodeShark_ [~androirc@40.135.239.174] has quit [Remote host closed the connection] 15:31 -!- aburan28 [~xypher@static-108-45-93-78.washdc.fios.verizon.net] has joined #bitcoin-core-dev 15:32 -!- MarcoFalke [50edb194@gateway/web/cgi-irc/kiwiirc.com/ip.80.237.177.148] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 15:32 -!- zooko [~user@2601:281:8001:26aa:dd9b:1012:6c2e:3d84] has joined #bitcoin-core-dev 15:33 -!- phantomcircuit [phantomcir@2600:3c01::f03c:91ff:fe73:6892] has joined #bitcoin-core-dev 15:50 -!- zmanian_ [uid113594@gateway/web/irccloud.com/x-ldinrjkhltfwqnoc] has quit [Ping timeout: 264 seconds] 15:52 -!- zmanian_ [uid113594@gateway/web/irccloud.com/x-ctiruyqstvnupaaf] has joined #bitcoin-core-dev 15:54 -!- Luke-Jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood] 15:54 -!- squidicuz [~squid@pool-173-48-117-206.bstnma.fios.verizon.net] has joined #bitcoin-core-dev 15:54 -!- phantomcircuit [phantomcir@2600:3c01::f03c:91ff:fe73:6892] has quit [Quit: quit] 15:55 -!- phantomcircuit [~phantomci@strateman.ninja] has joined #bitcoin-core-dev 15:55 -!- Luke-Jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-dev 16:12 -!- phantomcircuit [~phantomci@strateman.ninja] has quit [Quit: quit] 16:12 -!- phantomcircuit [phantomcir@2600:3c01::f03c:91ff:fe73:6892] has joined #bitcoin-core-dev 16:41 -!- PRab [~chatzilla@2601:40a:8000:8f9b:d59b:22a1:99bf:deb1] has quit [Quit: ChatZilla 0.9.92 [Firefox 41.0.2/20151014143721]] 17:17 -!- mcelrath [~mcelrath@38.121.165.30] has quit [Ping timeout: 252 seconds] 17:24 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-xshoveaqphmnqcxf] has quit [Quit: Connection closed for inactivity] 17:30 -!- zooko [~user@2601:281:8001:26aa:dd9b:1012:6c2e:3d84] has quit [Ping timeout: 246 seconds] 17:31 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has joined #bitcoin-core-dev 17:35 -!- belcher [~user@unaffiliated/belcher] has quit [Read error: Connection reset by peer] 17:36 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev 17:53 -!- phantomcircuit [phantomcir@2600:3c01::f03c:91ff:fe73:6892] has quit [Quit: quit] 17:58 -!- phantomcircuit [phantomcir@2600:3c01::f03c:91ff:fe73:6892] has joined #bitcoin-core-dev 18:09 -!- bsm1175321 [~mcelrath@38.121.165.30] has joined #bitcoin-core-dev 18:12 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 240 seconds] 18:40 -!- fanquake [~Adium@unaffiliated/fanquake] has joined #bitcoin-core-dev 19:02 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving] 19:33 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev 19:42 < Luke-Jr> a claim/report that running std::bad_alloc can cause a corrupt chainstate.. https://www.reddit.com/r/Bitcoin/comments/3rp0jb/another_dos_attack_how_to_safely_restart_node/ 19:43 -!- CodeShark [~CodeShark@2600:380:4b43:bae5:fd25:a062:1823:9dda] has quit [Ping timeout: 246 seconds] 19:45 -!- JackH [~Jack@host-80-43-141-3.as13285.net] has quit [Ping timeout: 250 seconds] 19:46 -!- CodeShark_ [~CodeShark@2600:380:4b43:bae5:fd25:a062:1823:9dda] has joined #bitcoin-core-dev 20:02 -!- PRab [~chatzilla@2601:40a:8000:8f9b:d59b:22a1:99bf:deb1] has joined #bitcoin-core-dev 20:09 < fanquake> jonasschnelli How easy is it for your nightly builder to spin up builds that contain multiple PRs? 20:32 < fanquake> Thinking #6931/2 + #6918 + #6954 + #6851 20:33 < dcousens> what is "Work Queue depth exceeded" coming from the bitcoin rpc 20:33 < dcousens> Its not JSON encoded either 20:33 < dcousens> Guessing something we didn't catch in libevent? 20:34 < dcousens> Seems to happen if I smash the RPC with a few hundred requests 20:34 < jgarzik> an internal libevent queue of requests waiting to be serviced by the rpc thread(s) 20:35 < fanquake> Look at the http request callback function in httpserver 20:36 < dcousens> Where is that limit set? 20:36 < jgarzik> basically a bunch of HTTPWorkItem's 20:36 < dcousens> also, wouldn't it be better to reply using application logic (aka, the RPC) rather than just "Work queue depth exceeded" 20:36 < jgarzik> int workQueueDepth = std::max((long)GetArg("-rpcworkqueue", DEFAULT_HTTP_WORK 20:37 < dcousens> ta, why isn't my search finding this 20:37 < dcousens> lol 20:37 < fanquake> Need to reindex :p 21:04 < gmaxwell> sdaftuar: FWIW, in memory UTXO during a IBD currently peaks out at 5486.7MB according to updatetip messages. 21:05 < sipa> gmaxwell: would be interested in what that number is with #6914 21:06 < gmaxwell> I'll benchmark a reindex next, and then a reindex with 6914 21:06 < sipa> great 21:06 < gmaxwell> currently waiting on stopnode.... turns out flushing a 5468 MB cache takes an awful long time. 21:07 < gmaxwell> about a minute and a half. 21:09 < gmaxwell> FWIW that sync from the network with no checkpoints took 3hrs 39 minutes. 21:09 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.] 21:20 < gmaxwell> https://people.xiph.org/~greg/utxo-memory-2015-11-06.png 21:20 < gmaxwell> :( 21:21 -!- guest234234 [~guest2342@223.207.239.102] has joined #bitcoin-core-dev 21:21 -!- guest234234 [~guest2342@223.207.239.102] has left #bitcoin-core-dev [] 21:22 -!- guest234234 [~guest2342@223.207.239.102] has joined #bitcoin-core-dev 21:38 -!- guest234234 [~guest2342@223.207.239.102] has quit [Ping timeout: 240 seconds] 22:01 -!- fanquake [~Adium@unaffiliated/fanquake] has quit [Quit: Leaving.] 22:08 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev 22:18 < dcousens> gmaxwell: IBD? 22:20 -!- ParadoxSpiral [~ParadoxSp@p508B8497.dip0.t-ipconnect.de] has joined #bitcoin-core-dev 22:22 < phantomcircuit> gmaxwell, libsecp? 22:23 < gmaxwell> dcousens: initial block download 22:23 < gmaxwell> phantomcircuit: yes. 22:24 < dcousens> gmaxwell: and before libsecp? 22:25 < phantomcircuit> "more" 22:25 < gmaxwell> probably also about 3 hours. 22:25 < gmaxwell> also with no signature validation.. also about three hours. 22:25 < gmaxwell> can't really make use of all the cpus on this host. 22:25 < dcousens> IO bound? 22:26 < gmaxwell> no, it's cpu bound in all the things that aren't signature validation. 22:26 < dcousens> haha 22:26 < gmaxwell> but thats what you get with over 8 cores or so. 22:28 < gmaxwell> (in particular it gets bound on the leveldb, block handing, hashing, etc. and that inserts bubbles in the downloading pipeline) 22:29 < phantomcircuit> oh no signature validation 22:29 < phantomcircuit> heh 22:31 < gmaxwell> phantomcircuit: no, this is with signature validation, specifically I'm testing the libsecp256k1 pull. 22:31 < gmaxwell> But if I bench with none it won't be much faster. 22:31 < gmaxwell> or openssl much slower. (because it'll just manage to use more of the cpus) 22:33 < phantomcircuit> ohh 22:35 < phantomcircuit> gmaxwell, it'll be a bit faster but yeah 22:35 -!- ParadoxSpiral [~ParadoxSp@p508B8497.dip0.t-ipconnect.de] has quit [Read error: Connection reset by peer] 22:36 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-dipefamoeiqhqgcr] has joined #bitcoin-core-dev 22:57 < wumpus> dcousens: the work queue is part of the HTTP server, happens before requests are handed off to the appliction logic (REST or RPC) 22:59 < dcousens> wumpus: all good :), just hadn't received it before, caught me off guard 23:01 < wumpus> don't know if the default number is high enough, do you hit it under quasi-normal circumstances? at some point under heavy 'abuse' it's better to reject clients instead of adding them to an overlong queue (also to prevents the requests from filling up memory), but the default of 16 may well be too conservative 23:11 < GitHub62> [bitcoin] MarcoFalke opened pull request #6958: [trivial] Cleanup maxuploadtarget (doc & log) (master...MarcoFalke-2015-maxupload) https://github.com/bitcoin/bitcoin/pull/6958 23:20 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev 23:21 -!- AtashiCon [arnavion@unaffiliated/arnavion] has quit [Quit: AtashiCon] 23:24 < phantomcircuit> gmaxwell, bleh it's really annoying nto having a MAC for the plaintext key 23:24 < phantomcircuit> any opposition to storing one? 23:31 -!- AtashiCon [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev 23:31 < wumpus> what plaintext key and MAC are you talking about? 23:31 * jonasschnelli is also wondering... 23:33 < phantomcircuit> oh the encrypted wallet entires 23:33 < phantomcircuit> entires* 23:33 < phantomcircuit> entries* 23:33 < phantomcircuit> the only way to tell which keys go to which master key (technically we support more than one... kind of) is to derive the public key 23:33 < wumpus> why do you want to change wallet encryption? it works fine 23:34 < phantomcircuit> it's very slow and corruption can only be detected when you unlock 23:34 < jonasschnelli> i think phantomcircuits optimizations makes sense 23:34 < phantomcircuit> we're basically using the private key -> public key derivation as a hash function right now 23:34 < phantomcircuit> which doesn't make a lot of sense 23:34 < wumpus> requiring an explicit pubkey derivation was on purpose, although due to AES padding trick you don't actually need tot do it 23:35 < wumpus> what do you want to optimize? what is not fast enough? 23:35 < wumpus> 'trying more passphrases per second'? :p 23:36 < gmaxwell> phantomcircuit: why do you need one for the plaintext key? you're checking the ciphertext. 23:36 < phantomcircuit> gmaxwell, the current check will detect whether you have multiple master keys mixed together into a single wallet file 23:36 < wumpus> what are we even trying to accomplish here? what is too slow that needs to be optimized? 23:37 < wumpus> I really don't like aimless optimization in the context of crypto code :) 23:37 < gmaxwell> wumpus: please see his existing PR, thie performance is not the main focus (though its quite slow with large wallets) 23:37 < gmaxwell> the existing behavior has no integrity checking at all for keys in an encrypted wallet, which is not very safe. 23:37 < phantomcircuit> wumpus, the main focus is detecting corruption on load instead of at the first unlock 23:38 < wumpus> what I want to avoid is to make it really fast but insecure 23:38 < gmaxwell> Nothing he's talking about would reduce security. 23:38 < wumpus> then hashing database records (eg, add a checksum to the data) will work well enough 23:38 < wumpus> no need to mess with the crypto 23:38 < gmaxwell> wumpus: Yes thats what his PR does. 23:39 < wumpus> ok, then what's the problem? 23:39 < gmaxwell> he's trying to also eliminate the need for the really slow check on unlock, it sounds like. (which is problamatically slow for wallets with very large numbers of keys) 23:40 < gmaxwell> the same is avoided already for non-encrypted wallets. 23:40 < wumpus> that never used to be the case, we only have to check one key/value pair 23:40 < wumpus> if you can assure the database is correct some other way 23:40 < wumpus> at some point someone wanted to check *all* keys on first unlock, as an integrity check 23:40 < wumpus> but that's not necessary if you add one at the db level 23:41 < phantomcircuit> that's what it does now :) 23:41 < wumpus> yeah, I heavily disagreed with that 23:41 < gmaxwell> wumpus: We _do_ check all of them on first unlock, as there is no other integrity check. 23:41 < wumpus> that indeed makes the first unlock slow on big wallets, that was clear, I even commented that back then 23:41 < phantomcircuit> gmaxwell, if we're OK with saying all the encrypted keys in the wallet have to be from the same master key 23:42 < gmaxwell> Sure, and thats still better than having no integrity check. 23:42 < phantomcircuit> then we dont need the hash on the plaintext key 23:42 < wumpus> anyhow, adding an integrity mechanism at the db level sounds good to me 23:42 < phantomcircuit> but then we should assert if there's more than 1 master key entry 23:42 < wumpus> but please don't change this code unnecessarily 23:44 < gmaxwell> phantomcircuit: does it actually work correctly now if there is more than one? 23:44 < phantomcircuit> gmaxwell, it correctly fails 23:44 < phantomcircuit> with the hashes it will silently not fail 23:45 < phantomcircuit> which would be double plus not good 23:45 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.] 23:45 < gmaxwell> ah so without the exhaustive test it won't know that some are encryted but with another key. 23:45 < wumpus> if you just add a checksum to the data, how can it add paths where it will silently work? you just add more checks, so more failures 23:46 < gmaxwell> wumpus: if he adds the checksum and turns off the exhaustive test (now not so important with the checksum) 23:46 < gmaxwell> phantomcircuit: so make it fail the load if there is more than one master key and call it done. 23:46 < wumpus> right 23:46 < wumpus> so that was the same in most actual releases (before the exhaustive check was added) 23:47 < wumpus> agreed 23:47 < phantomcircuit> sounds reasonable to me :) 23:47 < wumpus> at this point having more master keys *means* corruption 23:47 < wumpus> are you protecting all records or just the keys? 23:48 < wumpus> in any case, an assert to only have one master key is a good sanity check 23:48 < wumpus> who knows what happens with more, did anyone ever test that part... 23:49 < phantomcircuit> wumpus, it's unfortunately a huge nuisance to protect all the records 23:49 < phantomcircuit> i should look at doing that more carefully though... 23:50 < wumpus> phantomcircuit: ok, then doing just the keys is a good compromise 23:50 < wumpus> it is the most important data in the wallet 23:50 < gmaxwell> key records are already protected, ckey are not. but yea, most concerning is keys, and I think its a reasonable compromise. 23:51 < wumpus> if this was starting from scratch I'd want to do it for all data, but I can believe you if you say that becomes too ugly now 23:54 < phantomcircuit> hmm maybe it's not as hard as i was thinking 23:55 < wumpus> we should keep in mind that older versions should still be able to open the wallet 23:56 -!- guest234234 [~guest2342@223.207.239.102] has joined #bitcoin-core-dev 23:56 < phantomcircuit> wumpus, yeah i need to look at the serialization format to be sure 23:59 < jonasschnelli> does -debug=bench affect the performance in a recognizable way?