--- Day changed Thu May 12 2016 00:00 < gmaxwell> wumpus: this is on a haswell based xeon: 00:00 < gmaxwell> SHA256_avx,319,0.00344849,0.00346613,0.0034525 00:00 < gmaxwell> SHA256_basic,191,0.00531399,0.00533286,0.00532052 00:00 < gmaxwell> SHA256_rorx,351,0.00290081,0.00290608,0.002903 00:00 < gmaxwell> SHA256_rorx_x8ms,351,0.0028524,0.00286806,0.0028543 00:00 < gmaxwell> SHA256_sse4,319,0.00342679,0.00345755,0.003443 00:00 -!- xiangfu [~xiangfu@111.198.29.53] has quit [Ping timeout: 276 seconds] 00:00 -!- go1111111 [~go1111111@104.200.154.17] has quit [Ping timeout: 276 seconds] 00:01 -!- xiangfu [~xiangfu@111.198.29.53] has joined #bitcoin-core-dev 00:05 < gmaxwell> and a somewhat lower clocked broadwell-ep xeon: 00:05 < gmaxwell> SHA256_avx,255,0.0039705,0.0040791,0.00397483 00:05 < gmaxwell> SHA256_basic,175,0.00599706,0.00610304,0.00599851 00:05 < gmaxwell> SHA256_rorx,319,0.00334549,0.00345182,0.00334802 00:05 < gmaxwell> SHA256_rorx_x8ms,319,0.00328481,0.00339317,0.00328667 00:05 < gmaxwell> SHA256_sse4,255,0.00395852,0.00404716,0.00396052 00:07 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 00:11 -!- AaronvanW [~ewout@172pc231.sshunet.nl] has joined #bitcoin-core-dev 00:11 -!- AaronvanW [~ewout@172pc231.sshunet.nl] has quit [Changing host] 00:11 -!- AaronvanW [~ewout@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 00:13 -!- go1111111 [~go1111111@104.200.154.17] has joined #bitcoin-core-dev 00:18 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has quit [Remote host closed the connection] 00:18 < jonasschnelli> What does "OpenBlockFile failed for CBlockDiskPos(nFile=-1, nPos=0)" exactly mean? nFile=-1 looks after a undefined CDiskBlockPos? 00:18 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 00:19 < jonasschnelli> I'd like to debug the corruption I've got on my AARCH 00:20 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Remote host closed the connection] 00:32 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev 00:33 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Remote host closed the connection] 00:38 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Ping timeout: 250 seconds] 00:43 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 00:46 < jonasschnelli> wumpus: my results: 00:46 < jonasschnelli> SHA1,703,0.00148028,0.00152859,0.0015024 00:46 < jonasschnelli> SHA256_avx,415,0.00256598,0.00258584,0.0025782 00:46 < jonasschnelli> SHA256_basic,287,0.00375891,0.00400322,0.00383578 00:46 < jonasschnelli> SHA256_sse4,415,0.00255489,0.00261028,0.0025789 00:46 < jonasschnelli> SHA512,415,0.00251514,0.00256588,0.00252959 00:46 < jonasschnelli> Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz 00:48 < jonasschnelli> bench results of master... 00:48 < jonasschnelli> SHA1,703,0.00146294,0.00149019,0.00148072 00:48 < jonasschnelli> SHA256,255,0.00396442,0.00401562,0.00398446 00:48 < jonasschnelli> SHA512,415,0.00252354,0.0025878,0.00254187 00:48 < jonasschnelli> Ah. I guess SHA256_basic is the non-tweaked one.. 00:49 < jonasschnelli> So... looks after a 150% performance increase on my xeon server 00:49 < jonasschnelli> same on gmaxwell's results 00:52 < gmaxwell> interesting that the only thing the avx seems to do is hurt performance on AMD. :) 00:52 < gmaxwell> the rorx is somewhat faster though. 00:55 < jonasschnelli> What about the Intel SHA extentension? I Guess sse4 is a different thing. 00:55 < jonasschnelli> https://software.intel.com/en-us/articles/intel-sha-extensions 00:56 < jonasschnelli> I have a Xeon E3 and it seems that intels sha extension is only built into E5 and E7 families. 00:57 < gmaxwell> jonasschnelli: they don't exist yet. 00:58 < gmaxwell> (they were announced for broadwell then quietly dropped, I assume because the silicon failed) 00:58 < jonasschnelli> Also my "AArch64 Processor rev 4 (aarch64)" has Features : fp asimd aes pmull sha1 sha2 crc32 00:58 < jonasschnelli> mbedtls has some implementations for this: https://github.com/ARMmbed/mbedtls/pull/432/files 01:02 < jonasschnelli> After this: https://github.com/CriticalBlue/mbedtls/wiki it could give a x5 performance boost for SHA256 on ARM with that extension 01:03 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 252 seconds] 01:13 < GitHub135> [bitcoin] jonasschnelli opened pull request #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX (master...2016/05/fix_quit) https://github.com/bitcoin/bitcoin/pull/8046 01:13 < wumpus> thanks for the benchmark results! and yeah re SHA extensions: I'm trying to target hardware that actually exist in the wild :) 01:14 < wumpus> the AARCH64 sHA256 extension sounds reasonably interesting though, too bad Odroid C2 doesn't have it 01:16 < jonasschnelli> Yes. Not really an important thing. 01:17 < jonasschnelli> But enabling AVX should probably be something we should do. 01:17 < jonasschnelli> wumpus: Is the implementation directly copied from intel? 01:17 < wumpus> yes 01:18 < jonasschnelli> 1.5* performance boost in sha256 for advance vector enabled processors should be something we should try to get in soon. 01:19 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Quit: My Mac has gone to sleep. ZZZzzz…] 01:19 < jonasschnelli> I guess Sandy and Ivy supports it. 01:21 < wumpus> what you could try is see if it influences sync performance (a matter of adding CSHA256::SetImplementation(CSHA256::Impl::SSE4); in the inititalization) 01:23 < gmaxwell> s/AVX/SSE4/ 01:23 < jonasschnelli> SSE4 is ~the same performance boost on my Xeon. But right. We should probably go for AVX 01:23 < sipa> building 01:24 < gmaxwell> jonasschnelli: AVX is no faster on anything we've tested on and it hurts performance on Wumpus' AMD chip. 01:24 < jonasschnelli> gmaxwell: arg. Right. Mixed it up. 01:24 < gmaxwell> :) 01:24 < wumpus> yes, AVX seems useless 01:25 < gmaxwell> BlueMatt: can you benchmark this on skylake? 01:25 < sipa> SHA256_avx,255,0.00398244,0.00399226,0.00398497 01:25 < sipa> SHA256_basic,175,0.00608169,0.00611341,0.00608469 01:25 < sipa> SHA256_sse4,255,0.0039705,0.00398195,0.00397209 01:25 < jonasschnelli> wumpus: so I just pass in a CSHA256::SetImplementation(CSHA256::Impl::SSE4) in init.cpp somewhere? 01:25 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 01:26 < wumpus> jonasschnelli: yes, or alternatively change the default in src/crypto/sha256.cpp 01:26 < wumpus> sipa: what processor? 01:26 < sipa> Intel(R) Core(TM) i7-4800MQ, 2.6 GHz 01:26 < wumpus> thanks 01:27 < gmaxwell> sipa: thats another haswell. 01:28 < sipa> ok 01:28 < midnightmagic> I'm sure there's a reason for it, but is cpu detection and then dual codepaths, or even multiple-compile outputs not an option in the event a cpu family does benefit? 01:29 < midnightmagic> or has a significant benefit to these features just not been found i guess 01:29 < jonasschnelli> I could create a bench on a "2.6 GHz Intel Core i7" (guess this is Ivy Bridge) if it would be good to know. 01:30 < wumpus> midnightmagic: runtime CPU detection is necessary to integrate this, yes 01:30 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev 01:30 < wumpus> (right now I don't bother as this is just a hack/experiment) 01:30 -!- Cheeseo [~x@unaffiliated/cheeseo] has quit [Read error: Connection reset by peer] 01:31 -!- Cheeseo [~x@c-71-58-178-138.hsd1.pa.comcast.net] has joined #bitcoin-core-dev 01:31 -!- Cheeseo [~x@c-71-58-178-138.hsd1.pa.comcast.net] has quit [Changing host] 01:31 -!- Cheeseo [~x@unaffiliated/cheeseo] has joined #bitcoin-core-dev 01:31 < gmaxwell> midnightmagic: yes, it would be detected. But the test is important, e.g. it looks like we've learned not to use the AVX version even if supported. 01:31 < wumpus> in any case it's nice that sse4 gives the best speedup; that's the most common 01:31 * midnightmagic shakes fist at irritating icc dual code pathing which ((iirc) nullified it for commercial dev at prior employer 01:33 < gmaxwell> well rorx was clearly faster in my benchmarks. 01:34 < wumpus> but you actually have a cpu that supports it :) 01:34 < wumpus> oh! you did post your results, I missed that 01:35 < gmaxwell> I even highlit you. :) 01:35 < midnightmagic> uh. I have a lemote laptop sitting next to me in a backpack. in the event someone might like me to use it for testing, I'd be cool with helping. 01:36 < midnightmagic> (after I get it rebuilt) 01:38 < gmaxwell> wumpus: so does sipa and jonasschnelli. 01:39 < gmaxwell> They just didn't enable it. 01:45 < gmaxwell> IIRC rorx was introduced with haswell so it's a couple arch generations old now and pretty widely spread. though I can't say if the modest performance increase is really worth it. 01:50 < wumpus> ok added all of them to https://github.com/laanwj/bitcoin/tree/2016_05_sha256_accel 01:51 < wumpus> agreed, at the least we should only select one of the rorx variants, they're so similar in performance 01:51 < gmaxwell> sipa and jonasschnelli should test the rorx. (esp jonasschnelli since his chip is skylake) 01:53 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Quit: My Mac has gone to sleep. ZZZzzz…] 01:55 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 01:57 < jonasschnelli> A simple reindex comparison: 01:57 < jonasschnelli> Reindex up to block 200'000: 01:57 < jonasschnelli> SSE4: 389 seconds 01:57 < jonasschnelli> master: 406 seconds 01:58 < jonasschnelli> But I guess the SHA256 performance will be mostly "visible" during a synced state 01:59 < wumpus> at least when larger blocks come into play it may be more indicative, up to 200,000 are mostly easy blocks IIRC 01:59 < wumpus> but it's a noticable speedup that's good 02:00 < jonasschnelli> here my bench with rorx: 02:00 < jonasschnelli> SHA256_avx,351,0.00267656,0.00406031,0.00287071 02:00 < jonasschnelli> SHA256_basic,191,0.00403945,0.00774813,0.00572075 02:00 < jonasschnelli> SHA256_rorx,415,0.00223376,0.00340772,0.00260254 02:00 < jonasschnelli> SHA256_rorx_x8ms,319,0.00224853,0.00372237,0.00329514 02:00 < jonasschnelli> SHA256_sse4,255,0.00273502,0.00505805,0.00409666 02:00 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev 02:02 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Client Quit] 02:03 < wumpus> so sse4 139% rorx 173% 02:04 < jonasschnelli> Looks like... 02:05 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Quit: My Mac has gone to sleep. ZZZzzz…] 02:06 < jonasschnelli> I try now to compare a full reindex with std sha against rorx. 02:07 < wumpus> is this on the same Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz? 02:07 < jonasschnelli> Yes. 02:07 < wumpus> the numbers are fairly different that's why I ask 02:07 < jonasschnelli> Hmm... 02:08 < jonasschnelli> sse4 is extremely different.. right. 02:09 < jonasschnelli> Hmm.. maybe the reindex ran in the background... let me redo it. 02:09 < wumpus> min and max differences are also huge; possibly the benchmark framework should ignore the first run to 'prime the cache' 02:10 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 02:10 < wumpus> not that precise benchmarks are super-imporant here, it's clear that there is a win to using sse4/rorx 02:10 < jonasschnelli> SHA1,703,0.00148106,0.00254279,0.00150908 02:10 < jonasschnelli> SHA256_avx,415,0.00256918,0.00271057,0.00260393 02:10 < jonasschnelli> SHA256_basic,287,0.00377643,0.00395856,0.00385546 02:10 < jonasschnelli> SHA256_rorx,479,0.00214249,0.00227334,0.00217936 02:10 < jonasschnelli> SHA256_rorx_x8ms,479,0.00212789,0.00233448,0.00219144 02:10 < jonasschnelli> SHA256_sse4,383,0.00263,0.00277644,0.00266623 02:10 < jonasschnelli> SHA512,415,0.00252406,0.00267003,0.0025598 02:10 < wumpus> that looks better 02:11 < jonasschnelli> Yes. The reindex was probably running in the background. 02:11 < jonasschnelli> The latest bench was on a quite system (like the first ones without rorx i have posted) 02:14 < wumpus> these are comparable: https://github.com/laanwj/bitcoin/tree/2016_05_sha256_accel#intelr-xeonr-cpu-e3-1275-v5--360ghz 02:17 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Ping timeout: 246 seconds] 02:26 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 02:31 -!- xiangfu [~xiangfu@111.198.29.53] has quit [Ping timeout: 240 seconds] 02:36 -!- Ginnarr [~Ginnarr@unaffiliated/ginnarr] has joined #bitcoin-core-dev 02:37 -!- xiangfu [~xiangfu@111.198.29.53] has joined #bitcoin-core-dev 02:46 < GitHub120> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/69b3a6dd9d9a...5b736ddaa1c1 02:46 < GitHub120> bitcoin/master fad60b3 MarcoFalke: [qa] Fix bip9-softforks blockstore issue 02:46 < GitHub120> bitcoin/master 5b736dd Wladimir J. van der Laan: Merge #8041: [qa] Fix bip9-softforks blockstore issue... 02:46 < GitHub52> [bitcoin] laanwj closed pull request #8041: [qa] Fix bip9-softforks blockstore issue (master...Mf1604-qaBip9Blockstore) https://github.com/bitcoin/bitcoin/pull/8041 02:46 < GitHub57> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5b736ddaa1c1...2efe38b8323c 02:46 < GitHub57> bitcoin/master 3262316 Chirag Davé: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t... 02:46 < GitHub57> bitcoin/master 2efe38b Wladimir J. van der Laan: Merge #8004: signal handling: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t... 02:47 < GitHub29> [bitcoin] laanwj closed pull request #8004: signal handling: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t (master...fix_signal_handler) https://github.com/bitcoin/bitcoin/pull/8004 02:50 -!- MarcoFalke_ [8af6022a@gateway/web/freenode/ip.138.246.2.42] has joined #bitcoin-core-dev 02:52 -!- slackircbridge [~slackircb@45.55.41.36] has quit [Remote host closed the connection] 02:53 -!- slackircbridge [~slackircb@45.55.41.36] has joined #bitcoin-core-dev 02:57 < GitHub124> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2efe38b8323c...7c8558da362f 02:57 < GitHub124> bitcoin/master 8b0e497 Tyler Hardin: Qt: Add option to hide the system tray icon... 02:57 < GitHub124> bitcoin/master 7c8558d Wladimir J. van der Laan: Merge #8006: Qt: Add option to disable the system tray icon... 02:57 < GitHub140> [bitcoin] laanwj closed pull request #8006: Qt: Add option to disable the system tray icon (master...disable-tray) https://github.com/bitcoin/bitcoin/pull/8006 03:01 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Quit: My Mac has gone to sleep. ZZZzzz…] 03:08 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 03:13 < jonasschnelli> wumpus:do you have a tray icon menu on you Ubuntu? 03:14 < wumpus> yes, but it's messed up (ends up in the lower left of my screen), a known (upstream) issue with qt5 and ubuntu 14.04, so I'm thankful for the option to disable it 03:17 < wumpus> oh you mean if I can reproduce #8040 on ubuntu? I'll try 03:18 < wumpus> no, I can't: the tray icon only appears after the main window shows 03:19 < wumpus> it's not there during the splash screen 03:19 < wumpus> this would be an issue if we had a "launcher icon menu" 03:20 < wumpus> as the icon in the launcher appears as soon as the application loads 03:20 < jonasschnelli> wumpus: check https://github.com/bitcoin/bitcoin/issues/8043 03:20 < wumpus> then again that requires linking against ubuntu-specific libraries, so little chance 03:20 < jonasschnelli> I have this issue in 14.04 and 16.04 03:20 < MarcoFalke_> fedora core works fine 03:21 < wumpus> hmm, no I never had that problem 03:21 < MarcoFalke_> self-compiled, shows the tray icon after start 03:22 < wumpus> I've had this issue: https://github.com/bitcoin/bitcoin/issues/4826 and this issue: https://github.com/bitcoin/bitcoin/issues/5260 03:22 < wumpus> lots of fun with qt and the tray icon 03:22 < wumpus> the former doesn't seem to happen anymore though, so I think it was fixed upstream 03:23 < wumpus> the latter only happens with self-compiled bitcoin-qt against the system qt5, whichi s ancient 03:23 < wumpus> (5.2.1) 03:23 < wumpus> in any case not worrying. The missing menu is strange 03:24 < wumpus> adding the hide tray icon option is a good start, maybe at some point in the future it should be the default, too much trouble with it 03:31 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Quit: My Mac has gone to sleep. ZZZzzz…] 03:32 < jonasschnelli> Yes. That's what I was thinking. Though, the menu itself can be helpful. But the global menu space problem and the issues we always see with the tray menu make me think so. 03:32 -!- fengling [~fengling@111.198.29.53] has quit [Quit: WeeChat 1.4] 03:33 -!- PaulCapestany [~PaulCapes@204.28.124.82] has quit [Read error: Connection reset by peer] 03:33 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 03:34 < wumpus> the right way to do this on ubuntu would be the 'launcher menu' route. But due to the practical concern with distributing GUI binaries on linux that'd only work, at most, for the ppa as that is custom-built for ubuntu 03:34 -!- PaulCapestany [~PaulCapes@204.28.124.82] has joined #bitcoin-core-dev 03:39 < wumpus> in any case I'll try on ubuntu 16.04 too and see if I can reproduce 8043 03:42 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 03:44 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 03:53 < jonasschnelli> wumpus: thanks. 04:01 -!- achow101 [~achow101@pool-96-227-114-115.phlapa.fios.verizon.net] has joined #bitcoin-core-dev 04:14 < GitHub176> [bitcoin] MarcoFalke opened pull request #8047: [qa] test_framework: Set wait-timeout for bitcoind procs (master...Mf1605-qaUtilTimeout) https://github.com/bitcoin/bitcoin/pull/8047 04:16 < GitHub130> [bitcoin] jonasschnelli pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7c8558da362f...169d379c9835 04:16 < GitHub130> bitcoin/master 34ebceb Jonas Schnelli: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX 04:16 < GitHub130> bitcoin/master 169d379 Jonas Schnelli: Merge #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX... 04:16 < GitHub15> [bitcoin] jonasschnelli closed pull request #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX (master...2016/05/fix_quit) https://github.com/bitcoin/bitcoin/pull/8046 04:18 -!- MarcoFalke_ [8af6022a@gateway/web/freenode/ip.138.246.2.42] has quit [Quit: Page closed] 04:26 -!- cryptapus [~cyptapus@87.254.202.201] has joined #bitcoin-core-dev 04:26 -!- cryptapus [~cyptapus@87.254.202.201] has quit [Changing host] 04:26 -!- cryptapus [~cyptapus@unaffiliated/cryptapus] has joined #bitcoin-core-dev 04:32 -!- Ginnarr [~Ginnarr@unaffiliated/ginnarr] has quit [Quit: Textual IRC Client: www.textualapp.com] 04:50 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 04:50 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 04:51 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 04:53 < GitHub194> [bitcoin] laanwj opened pull request #8048: doc: Remove outdated qt4 install information from README.md (master...2016_05_doc_noqt4) https://github.com/bitcoin/bitcoin/pull/8048 04:58 -!- murch [~murch@p4FE3AC9B.dip0.t-ipconnect.de] has joined #bitcoin-core-dev 05:11 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Quit: My Mac has gone to sleep. ZZZzzz…] 05:11 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 05:16 -!- molz [~molly@unaffiliated/molly] has quit [Read error: Connection reset by peer] 05:17 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 05:23 -!- murch1 [~murch@p4FE3B1CE.dip0.t-ipconnect.de] has joined #bitcoin-core-dev 05:24 -!- jtimon [~quassel@65.28.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev 05:25 -!- murch [~murch@p4FE3AC9B.dip0.t-ipconnect.de] has quit [Ping timeout: 244 seconds] 05:27 < GitHub159> [bitcoin] laanwj opened pull request #8049: Expose information on whether transaction relayed is enabled in `getnetwork` (master...2016_05_rpc_relaytxes) https://github.com/bitcoin/bitcoin/pull/8049 05:28 < GitHub116> [bitcoin] laanwj closed pull request #7841: Expose information on whether transaction relayed is enabled in getnetwork (master...patch-2) https://github.com/bitcoin/bitcoin/pull/7841 05:33 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 05:44 -!- xiangfu [~xiangfu@111.198.29.53] has quit [Ping timeout: 276 seconds] 05:45 -!- xiangfu [~xiangfu@49.4.170.106] has joined #bitcoin-core-dev 06:08 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Quit: My Mac has gone to sleep. ZZZzzz…] 06:08 -!- gill3s [~textual@unaffiliated/gill3s] has joined #bitcoin-core-dev 06:10 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has quit [Remote host closed the connection] 06:10 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 06:12 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has quit [Remote host closed the connection] 06:12 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 06:20 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has quit [Remote host closed the connection] 06:20 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 06:38 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 250 seconds] 06:40 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 06:40 -!- TomMc [~tom@unaffiliated/tommc] has joined #bitcoin-core-dev 06:42 -!- zooko [~user@2601:281:8000:8387:cdca:602d:94cb:bd62] has quit [Ping timeout: 250 seconds] 06:44 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 06:48 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has quit [Remote host closed the connection] 06:48 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 06:50 < BlueMatt> gmaxwell: still need it benchmarked? 06:50 < gmaxwell> BlueMatt: we got skylake numbers from jonasschnelli 06:52 < BlueMatt> kk 06:54 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 06:59 -!- BonyM [~BonyM-I@ua-83-227-211-4.cust.bredbandsbolaget.se] has quit [Quit: Leaving] 07:00 -!- gill3s [~textual@unaffiliated/gill3s] has quit [Read error: Connection reset by peer] 07:02 -!- BonyM [~BonyM-I@ua-83-227-211-4.cust.bredbandsbolaget.se] has joined #bitcoin-core-dev 07:02 -!- MarcoFalke [8af602fa@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.250] has joined #bitcoin-core-dev 07:14 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has joined #bitcoin-core-dev 07:29 < jonasschnelli> wumpus: benchmark results of full reindex up to height 400'000 07:29 < jonasschnelli> master, std sha: 8'431 seconds 07:29 < jonasschnelli> rorx: 7'836 07:31 -!- xiangfu [~xiangfu@49.4.170.106] has quit [Ping timeout: 244 seconds] 07:35 -!- xiangfu [~xiangfu@111.198.29.53] has joined #bitcoin-core-dev 07:37 < instagibbs> jonasschnelli, what is this line doing? https://github.com/bitcoin/bitcoin/pull/8035/files#diff-fcc78df4b7178e5b09f83f38196fef8bR59 07:38 < instagibbs> I suppose I don't get the serialization code well enough 07:38 -!- G1lius [~stefangil@85.17.25.22] has joined #bitcoin-core-dev 07:46 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev 07:48 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev 07:49 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Client Quit] 07:52 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)] 07:59 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has quit [Remote host closed the connection] 07:59 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 08:09 -!- MarcoFalke [8af602fa@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.250] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 08:10 -!- earlest [~denetrabu@96.93.57.150] has joined #bitcoin-core-dev 08:13 -!- muuqwaul [~denetrabu@96.93.57.150] has quit [Ping timeout: 240 seconds] 08:15 < jonasschnelli> instagibbs: i think this line is not required. I guess I copied this from another class. Will have a look soon. 08:15 -!- zooko [~user@c-24-9-79-61.hsd1.co.comcast.net] has joined #bitcoin-core-dev 08:15 < instagibbs> ok, that's my only ? left. Thanks 08:16 < luke-jr> it's doing something there 08:16 < luke-jr> nVersion is the method param, not the class var 08:16 < luke-jr> I think READWRITE uses it? 08:16 < luke-jr> definitely needs better docs 08:17 < instagibbs> ... what 08:17 < luke-jr> + inline void SerializationOp(Stream& s, Operation ser_action, int nType, int nVersion) 08:17 < instagibbs> oh 08:17 < instagibbs> right 08:18 < luke-jr> I suggest renaming one of the nVersions 08:18 < luke-jr> as-is, this is obfuscated code 08:18 < instagibbs> agreed 08:18 < jonasschnelli> Yes. Will rename it. 08:18 < jonasschnelli> I think I took it 1:1 from CKeyMetadata 08:21 -!- zooko` [~user@50.141.118.140] has joined #bitcoin-core-dev 08:22 -!- zooko [~user@c-24-9-79-61.hsd1.co.comcast.net] has quit [Ping timeout: 240 seconds] 08:28 -!- zooko`` [~user@c-24-9-79-61.hsd1.co.comcast.net] has joined #bitcoin-core-dev 08:30 -!- zooko` [~user@50.141.118.140] has quit [Ping timeout: 250 seconds] 08:30 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Remote host closed the connection] 08:31 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 08:35 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Ping timeout: 250 seconds] 08:44 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 08:45 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev 08:48 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 09:04 -!- bysherper [~denetrabu@96.93.57.150] has joined #bitcoin-core-dev 09:07 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has quit [] 09:07 -!- earlest [~denetrabu@96.93.57.150] has quit [Ping timeout: 240 seconds] 09:10 -!- zooko`` [~user@c-24-9-79-61.hsd1.co.comcast.net] has quit [Ping timeout: 246 seconds] 09:14 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Quit: laurentmt] 09:24 -!- xiangfu [~xiangfu@111.198.29.53] has quit [Remote host closed the connection] 09:28 -!- earlest [~denetrabu@96.93.57.150] has joined #bitcoin-core-dev 09:31 -!- bysherper [~denetrabu@96.93.57.150] has quit [Ping timeout: 240 seconds] 09:35 -!- Don_John [~Don@247-223-114-134.nat.resnet.nau.edu] has joined #bitcoin-core-dev 09:35 -!- Don_John [~Don@247-223-114-134.nat.resnet.nau.edu] has quit [Remote host closed the connection] 09:45 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 09:49 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 09:58 -!- zooko [~user@sta-207-174-117-102.rockynet.com] has joined #bitcoin-core-dev 09:58 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev 09:59 -!- PRab_ [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has joined #bitcoin-core-dev 10:02 -!- G1lius [~stefangil@85.17.25.22] has quit [] 10:03 -!- cryptapus_ [~cyptapus@unaffiliated/cryptapus] has joined #bitcoin-core-dev 10:03 -!- PRab [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has quit [Ping timeout: 240 seconds] 10:03 -!- PRab_ is now known as PRab 10:05 -!- PRab [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has quit [Read error: Connection reset by peer] 10:05 -!- PRab [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has joined #bitcoin-core-dev 10:05 -!- cryptapus_ [~cyptapus@unaffiliated/cryptapus] has quit [Client Quit] 10:06 -!- cryptapus_ [~cyptapus@87.254.202.201] has joined #bitcoin-core-dev 10:06 -!- cryptapus_ [~cyptapus@87.254.202.201] has quit [Changing host] 10:06 -!- cryptapus_ [~cyptapus@unaffiliated/cryptapus] has joined #bitcoin-core-dev 10:06 -!- cryptapus [~cyptapus@unaffiliated/cryptapus] has quit [Read error: Connection reset by peer] 10:07 -!- PRab_ [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has joined #bitcoin-core-dev 10:10 -!- PRab [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has quit [Ping timeout: 246 seconds] 10:10 -!- PRab_ is now known as PRab 10:10 -!- PRab [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has quit [Read error: Connection reset by peer] 10:12 -!- PRab_ [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has joined #bitcoin-core-dev 10:15 -!- PRab_ is now known as PRab 10:22 -!- PRab [~chatzilla@c-68-55-113-152.hsd1.mi.comcast.net] has quit [Ping timeout: 276 seconds] 10:27 -!- TomMc [~tom@unaffiliated/tommc] has quit [Ping timeout: 252 seconds] 10:29 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 10:41 -!- TomMc [~tom@unaffiliated/tommc] has joined #bitcoin-core-dev 10:42 -!- cryptapus_ is now known as cryptapus 10:55 -!- jcorgan [~jcorgan@unaffiliated/jcorgan] has joined #bitcoin-core-dev 10:59 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev 11:14 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 244 seconds] 11:17 < wumpus> jonasschnelli: nice, that's quite an improvement 11:18 < jonasschnelli> wumpus: Yes. I guess most reindex work in ECDSA and not SHA. So yes. It's an impressive improvement. 11:18 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 11:18 < jonasschnelli> I guess during a synced state the improvements are even more visible. 11:19 < jcorgan> are there still thursday weekly dev meetings here? 11:19 < wumpus> yes 11:19 < helo> yes 11:19 < gmaxwell> not for another 45 minutes. 11:19 < jcorgan> ah, off by an hour 11:21 -!- zooko [~user@sta-207-174-117-102.rockynet.com] has quit [Ping timeout: 260 seconds] 11:39 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Ping timeout: 276 seconds] 11:40 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has joined #bitcoin-core-dev 11:42 < droark> Hi. Question about PR #7972. It adds a new file (qa/rpc-tests/create_cache.py) that creates a cache used, I assume, to assist with parallelizing the qa/rpc-tests code. I don't quite understand this. Can somebody help me out? TIA. 11:44 -!- bysherper [~denetrabu@96.93.57.150] has joined #bitcoin-core-dev 11:44 < jonasschnelli> droark: You might need to ask MarcoFalke. But IIRC, you need to create the cache before you run the script in parallel because you run into the risk of different cache versions... 11:44 < sdaftuar> jonasschnelli: droark: right, i assume it's to avoid having two jobs try to do it at the same time 11:46 < droark> Sounds reasonable. What exactly is being cached, though? The results? The runtime environment for each instance? Something else? 11:46 -!- murch1 is now known as murch 11:46 < sdaftuar> droark: it's a cached blockchain 11:46 -!- earlest [~denetrabu@96.93.57.150] has quit [Ping timeout: 240 seconds] 11:46 < sdaftuar> take a look at initialize_chain in qa/rpc_tests/test_framework/util.py 11:47 < droark> Ahhh. Thanks! I'll ping Marco and double check the code but it sounds right. 11:48 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 11:48 < lclc> Is anyone of the devs who are coming to Zurich arriving the day before (Thursday) ? 11:49 < sipa> i'll be there on thursday evening 11:50 < jcorgan> i get there thu evening as well 11:50 < lclc> That doesn't surprise me :D I'm asking because I'd like to organize a meetup the evening before so the community also has something from the event :) 11:50 < jcorgan> well, 6pm at the airport 11:51 < lclc> Would be cool if someone would give a talk - topic is up to you. Our meetups are 30-110 people, free to attend, and recorded (www.bitcoinlectures.tv - YT Channel) 11:52 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev 11:52 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 12:00 < wumpus> I'm also arriving in zurich thursday evening 12:00 < sdaftuar> meeting time? 12:01 < wumpus> #meetingstart 12:01 < petertodd> yup 12:01 < wumpus> #startmeeting 12:01 < lightningbot> Meeting started Thu May 12 19:00:44 2016 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot. 12:01 < lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic. 12:01 < BlueMatt> topics? 12:01 < wumpus> from last week: 12:02 < wumpus> add a file bip-0009/assignments.md in the bips repository: btcdrak made a pull for that 12:02 < wumpus> discuss testnet activation date on bitcoin-dev mailing list 12:02 < wumpus> ead bluematt's compact block bip 12:02 < wumpus> +r 12:02 -!- jannes [~jannes@178.132.211.90] has quit [Quit: Leaving] 12:03 < wumpus> #link https://github.com/bitcoin/bips/pull/386 12:03 < wumpus> any other topic proposals? 12:03 < luke-jr> on that note, it'd be nice if people didn't ACK/NACK bips they are not a listed author of ;) 12:04 < sipa> was segwit testnet discussed on the ml? 12:04 < kanzure> luke-jr: please elaborate. 12:04 < BlueMatt> re: compact block, got good feedback from a few, made some slight tweaks but need to finish tweaks in the coming days, so no need to discuss here, I think? 12:05 < luke-jr> kanzure: when it comes to a BIP draft, the only ACK/NACK that matters is the Author(s); others posting ACK/NACK just makes me spend time looking at the Author header to confirm I need to ignore it 12:05 < wumpus> well in the case of #386 I felt I had the right to give my opinion because I was one of the people to come up with the idea last meeting, even though I'm not listed as author 12:05 < sipa> i guess the question is where BIPs are to be discussed 12:05 < luke-jr> sipa: bitcoin-dev ML 12:06 < sipa> and the idea is that that would be the mailinglist 12:06 < sipa> but... 12:06 < kanzure> luke-jr: wouldn't NACKs from non-authors still be useful? why should the authors be the only trusted source of updates ? i don't understand. 12:06 < instagibbs> oh, the compact blocks got a bip# 12:06 < paveljanik> I don't understand either. If the author dies in accident, we can no longer change the BIP? or what? 12:06 < morcos> luke-jr: i'd suggest that you just come up with a way to distinguish. for instance ACK/NACK shoudl be interpreted by the author and the author shoudl say Approved or Ready or something for your knowledge on when to merge 12:06 < kanzure> luke-jr: i'm willing to believe you're right but you need better reasons yo. 12:06 < wumpus> I agree that ack/nack isn't very useful, in comparison to more detailed comments 12:06 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Quit: laurentmt] 12:07 < petertodd> kanzure: if I were author of a BIP an ACK would give me confidence; a NACK could be useful criticism 12:07 < sipa> paveljanik: an active BIP can't be modified anyway 12:07 < kanzure> wumpus: right, sure. 12:07 < luke-jr> kanzure: the BIP is a document by the Author(s). ideally, they should have exclusive merge access for changes, but GitHub doesn't support that 12:07 < paveljanik> text clarification, etc... 12:07 < kanzure> okay this is just trivial though, luke-jr isn't going to stop me from ACKing on BIPs i didn't author :) 12:07 < morcos> sipa: in some cases it should be, if for instance the BIP wording is incorrect ala 34 12:07 < luke-jr> paveljanik: BIP 1 technically allows me to reassign BIP Author in some cases IIRC 12:07 < sipa> morcos: yeah - makes sense 12:07 < jonasschnelli> We should also define a rule how we deal with links to implementations that have implemented the BIP. 12:08 < sipa> jonasschnelli: yes 12:08 < jonasschnelli> IMO we should not add any implementation links. 12:08 < jonasschnelli> (expect of sample impl.) 12:08 < wumpus> many bips have a list of implementations; what's wrong with that? 12:08 < jonasschnelli> Its just noisy 12:08 < sipa> wumpus: in BIP32 there are continuously pull requests to add links 12:08 < gmaxwell> it gets flooded with spammy updates. 12:08 < wumpus> I think it can be useful to have e.g. implementations in many languages 12:08 < jonasschnelli> And we don't control the implementations I guess 12:08 < sipa> wumpus: which imho function as nothing more than advertizement 12:08 < kanzure> and then we have to moderate the additions 12:08 < jonasschnelli> sipa: Yes! 12:08 < gmaxwell> And if we're not looking at it, we're eventually going to get a malware example BIP32 impl. :) 12:08 < wumpus> ok... 12:08 < luke-jr> BIP 2 comments would be a nice place to list implementations, but that was controverisal 12:09 < jonasschnelli> I could imaging more and more advertising like PR to come up in future. 12:09 < wumpus> I think the requirement should at least be to point to the code 12:09 < wumpus> not just the product 12:09 < paveljanik> so what is the topic now? ;-) 12:09 < wumpus> to prevent advertizement 12:09 < kanzure> i think the topic was "feedback about compact block relay BIP things" 12:10 < jonasschnelli> Which still intersects (a little bit) 12:10 < jcorgan> it is useful, however, when reading a BIP, to at least get pointed to reference code, but it need not be every implementation that anyone wants to list thereafter 12:10 < wumpus> I think the topic is how to handle BIP implementations, which is as good a topic as anything 12:11 < wumpus> but then you get the fight about what is reference code and what is not 12:11 < jonasschnelli> Links are always difficult. You need to check them... then what if a link diverges from the BIP specification? 12:11 < jonasschnelli> Do we check it? 12:11 < wumpus> in some cases it's clear if the BIP author wrote code themselves 12:11 < kanzure> yeah another threat vector is someone selling their github account in the future 12:11 < kanzure> and then a BIP links to someone's github 12:11 < jonasschnelli> Yes. These are the link we should keep there... 12:11 < jcorgan> or the BIP author themselves pick one or more... 12:11 < gmaxwell> If it's upfront then we can trust the bip author and review process to have had some chance to verify it. 12:12 < wumpus> yes, I'd say it's up to the BIP author 12:12 < wumpus> like other changes to the BIP 12:12 < kanzure> other problem is keeping track of upstream updates like security fixes, oops. anyway, this is a lot of work. 12:12 < kanzure> we should include hashes of the reference source code, in the BIP text 12:12 < luke-jr> maybe BIP PRs should be PGP signed 12:12 < wumpus> this is not something that should be globally decided 12:12 < gmaxwell> I could see the criteria being different for different BIPs. 12:12 < jonasschnelli> wumpus: Yes. This makes sense. 12:12 < kanzure> by hashing the source code we can at least have a way for readers to verify that the source code is still the part we meant to hyperlink to :) 12:12 < wumpus> gmaxwell: exactly 12:12 < wumpus> I mean the author of bip32 could say 'enough is enough' 12:13 < jcorgan> er, link to a URL and commit hash? 12:13 < wumpus> some other BIPs have far fewer implementations and the author may be happy to see one 12:13 < gmaxwell> (unfortunately, it's BIP32 that gets 95% of this-- key generation is an especially awkward place for random found on the internet code) 12:13 < petertodd> my BIP65 links to two separate demo repos that just give some sample code, which is probably something we can generally consider as acceptable (ignoring the issue of more complex implementations that aren't pure demos) 12:13 < jonasschnelli> I like jcorgan idea to link to sourcecode-only with a commit hash 12:13 < jonasschnelli> But kinda static. 12:13 < wumpus> Itend to agree with that. Link to the actual code, not the product 12:14 < wumpus> not 'blawallet implements bip32', no, *the code linked here shows how we implemented BIP32 in language Z* 12:14 < sipa> wumpus: agree 12:14 < luke-jr> +1 12:14 < petertodd> wumpus: +1 12:15 < paveljanik> yes 12:15 < jonasschnelli> ack 12:15 < wumpus> ok 12:15 < kanzure> we should review existing bips for source code links that do not include a commit hash. branch names are not OK. 12:15 < kanzure> i mean, branch names without a commit hash are not OK. 12:15 < wumpus> other topics? 12:15 < paveljanik> Revies Jonas' HD - #8035 12:16 < jonasschnelli> I have a tiny topic proposal that is very into the impl. teretorry.. 12:16 < jonasschnelli> RPC long poll notifications 12:16 < kanzure> have we received an overview from sipa yet about areas of segwit that he feels should be most thoroughly reviewed 12:16 < sipa> kanzure: no, sorry 12:16 < kanzure> can we get 10 volunteers to heckle sipa about this? 12:16 < sipa> thanks for the reminder 12:16 < wumpus> #action Revies Jonas' HD - #8035 12:17 < jonasschnelli> https://github.com/bitcoin/bitcoin/pull/8035 ... very simple and easy to review 12:17 < paveljanik> jonasschnelli, can you add some after-merge TODO list there? 12:17 < jonasschnelli> I'm happy to add some RPC tests... 12:17 < jonasschnelli> There is no required after-merge PR... thats the nice thing. 12:18 < jonasschnelli> But most important we should enable flexible bip32 keypath 12:18 < paveljanik> import, dump... at least... 12:18 < instagibbs> jonasschnelli, a bit hard to rpc test without import/dump? (perhaps offline discussion) 12:18 < jonasschnelli> There are concerns with importing single keys into the bip32 wallet... 12:18 < jonasschnelli> They would not be covered by "a old backup" 12:19 < wumpus> but that's kind of obvious :) 12:19 < btcdrak> maybe need a sweep feature. 12:19 < wumpus> yes, a sweep feature would be nice 12:19 < jonasschnelli> There are plenty of possible features... 12:19 < jonasschnelli> But because of the lack of reviewers,.. we need babysteps 12:19 < sipa> let's not discuss all possible features 12:19 < luke-jr> sweep would be nice, but non-trivial (currently no way to iterate over the UTXO set I think) 12:19 < sipa> just what is necessary to work and test 12:19 < jonasschnelli> agree with sipa. 12:20 < jonasschnelli> Just review and ack it! :P 12:20 < luke-jr> so what about RPC longpoll? 12:20 < wumpus> I agree 12:20 < sipa> if import/export is necessary for testing, then maybe some functionality for that is warranted 12:20 < wumpus> scope creep agian 12:20 < instagibbs> well, it only came up due to testing 12:21 < wumpus> luke-jr: https://github.com/bitcoin/bitcoin/pull/7949 12:21 < jonasschnelli> RPC long poll would would allow remote GUI and remote wallet with a very easy setup. IMO it could allow third party developers to write simple remote GUIS, web-frontends, etc. 12:22 < jonasschnelli> I would even say RPC long poll has more potential then ZMQ for core. 12:22 < wumpus> I'm kind of divided on the notification thing to be honest - I'd prefer to stick to only one notification mechanism in bitcoin core (apart from the silly -Xnotify), and somehow zmq got a precedent there 12:23 < luke-jr> calling pollnotifications myself seems like it would disrupt an application relying on it? 12:23 < jonasschnelli> I also don't like multiple notification channels. 12:23 < luke-jr> ie, we need channels 12:23 < jonasschnelli> But how would you connect a remote GUI over the internet... 12:23 < luke-jr> wumpus: zmq is crazy though :< 12:23 < luke-jr> also note we already have longpolling 12:23 < wumpus> yes it should definitely have multiple channels, the current code supports only one client 12:23 < jonasschnelli> wumpus: no 12:23 < wumpus> luke-jr: where were you to NACK zmq when it was added? 12:23 < jcorgan> i'm happy to look at improving zmq 12:23 < luke-jr> jonasschnelli: no RPC over internet ever :< 12:23 < jonasschnelli> I have added support for clients 12:24 < luke-jr> wumpus: just because zmq is crazy doesn't mean optional ZMQ support should be excluded .. 12:24 < jonasschnelli> RPC over internet over a reverse proxy is a wide use pratice. 12:24 < wumpus> (I don't think zmq is necessarily crazy) 12:24 < jtimon> sorry, what are the complaints with zmq? it is optional anyway 12:24 < wumpus> jtimon: I'm not sure either, it seems to be fashionable to complain about it 12:24 < luke-jr> jtimon: it's not optional if it is an excuse to exclude other systems 12:24 < luke-jr> zmq breaks protocol compatibility for minor bumps 12:24 < jonasschnelli> Working towards decoupling of the GUI and the wallet requires well defined channels/APIs 12:24 < luke-jr> ie, 4.1 won't work with 4.0 right 12:25 < jonasschnelli> ZMQ would require a tunnel (VPN, stunnel, etc.). 12:25 < luke-jr> also see all the Elements functionary problems due to ZMQ 12:25 < jcorgan> ZMQ 4.x is implementing CurveCP 12:25 < jtimon> I think it's a wonderful tool I would like to use more (although maybe the fact that its creator rewrote it completely in nanomsg may indeed indicate that some parts of zmq have become too complex) 12:25 < wumpus> zmq is basically only useful for local system, but so is RPC, it's not meant to be used over the internet 12:26 < jcorgan> see comment about curvecp 12:26 < wumpus> if you write tunneling for RPC - why not tunnel the notifications too? 12:26 < jonasschnelli> I just think Core users would love to have a GUI/wallet-frontend that can run on a different machine. 12:26 < jcorgan> http://curvezmq.org/ 12:26 < jtimon> wumpus: exactly, it's for your intranet or at most inside a vpn (although I haven't tried that myself) 12:26 < luke-jr> jonasschnelli: yes, for years now that has been desirable 12:26 < wumpus> jonasschnelli: I agree, but does that need RPC notifications? 12:26 < jonasschnelli> wumpus: RPC does work extremly well in reverse-proxy mode. 12:26 < wumpus> jonasschnelli: what would you use it for? 12:27 < luke-jr> wumpus: right now it requires polling 12:27 < wumpus> e.g. to get notification of transactions/blocks the P2P protocol works fine 12:27 < jonasschnelli> Look at rtorrent or any other RPC daemon software. 12:27 < jonasschnelli> Polling is extremely inefficient. Long polling would allow clients to get "realtime" events while not require any other dependencies... 12:27 < jonasschnelli> And the code changes are tiny... 12:27 < wumpus> yes, agreed jonasschnelli 12:28 < wumpus> yes the code changes are tiny 12:28 < jonasschnelli> You could setup a save and secure remote bitcoind with a single apache/nginx config. 12:28 < wumpus> but I'm a bit afraid the same will happen as with REST, people will want to pile up things on top, and now for every notification people will want zmq and rpc longpull support 12:28 < jonasschnelli> Now do the same with ZMQ notifications... :) 12:29 < wumpus> just like people want every possible data item both through RPC and REST 12:29 < sipa> well ZMQ/P2P are suboptimal to write a remote GUI, as you can't filter for just wallet transactions 12:29 < wumpus> I believe thatthere is value to keeping core limited to one interface 12:29 < luke-jr> wumpus: I don't see why this is a problem 12:29 < jonasschnelli> Yes. If i had a blank paper. I would drop ZMQ and REST in favor or RPC longpoll and the normal RPC. 12:29 < jcorgan> longpoll does indeed solve some of my original motivation for adding ZMQ 12:29 < sipa> jonasschnelli: i think you're biased because you're only thinking about one use case 12:29 < wumpus> well my iniitial idea for notifications was also something like longpoll, or a streaming socket 12:30 < wumpus> then a proxy from that to zmq 12:30 < wumpus> but zmq was first 12:30 < luke-jr> zmq is optional 12:30 < luke-jr> someone shouldn't need to install zmq for notifications 12:30 < jtimon> zmq's req/rep could replace both rpc and rest 12:30 < jonasschnelli> There is one big advantage of RPC long polling... 12:30 < jonasschnelli> you can have private notifications... 12:30 < jonasschnelli> secured behind auth 12:30 < wumpus> do we want private notifications? 12:30 < jonasschnelli> Like a new relevant wallet tx comes in 12:31 < sipa> wumpus: for a remote wallet gui you do 12:31 < jonasschnelli> It would be on the same level then the RPC security... 12:31 < jonasschnelli> just instead of poll you can have push 12:31 < wumpus> connect a remote wallet GUI through RPC? 12:31 < wumpus> wasn't RPC meant to be for local usage? 12:31 < jonasschnelli> I would not add this over ZMQ because of the unknown risks 12:31 < luke-jr> jonasschnelli: btw FYI https://en.bitcoin.it/wiki/Wallet_protocol 12:31 < jonasschnelli> wumpus: that is an argument. 12:31 < wumpus> I think the idea was to attach a *wallet*, not a wallet GUI 12:32 < wumpus> the wallet needs to be split from the core 12:32 < luke-jr> a wallet can be attached over p2p fine 12:32 < jonasschnelli> Another solution would be to provide a tiny daemon that would interact between core <-> remote GUI/wallet. 12:32 < wumpus> I'd prefer for core to handle semi-public data, then have a more restricted wallet 12:32 < jonasschnelli> But that tiny daemon would speak RPC with the outside/internet. 12:32 < luke-jr> ideally we should have node <-> wallet <-> GUI 12:33 < wumpus> luke-jr: yes 12:33 < jonasschnelli> luke-jr: But how would you see the communication channel between wallet <> GUI? 12:33 < jcorgan> another motivation for adding ZMQ was to allow bitcoind to be a trusted "border gateway", that spoke P2P and consensus, and then stuff behind it would be very simple ZMQ-based software that didn't need to know all about those things 12:33 < wumpus> but wallet<->GUI could be a competely different protocol than node<->wallet 12:33 < jonasschnelli> Needs to be bidirect. 12:33 < luke-jr> jonasschnelli: see wiki link; or something like what you're doing 12:33 < jonasschnelli> node <> wallet is probably p2p? 12:33 < sipa> yes 12:33 < luke-jr> yes 12:34 < morcos> node <-> wallet being p2p leaves a lot of missing pieces 12:34 < wumpus> maybe authentiated P2P 12:34 < morcos> fee estimation 12:34 < jonasschnelli> But what direction do we want to go for <-> ? 12:34 < wumpus> as you proposed 12:34 < wumpus> with some private extensions 12:34 < sipa> jonasschnelli: up to the wallet 12:34 < morcos> mempool actions (eviction, how close to top of mempool, whether it was accepted) 12:34 < jonasschnelli> morcos: fee estimations can be done with authentication/private extensions. 12:34 < BlueMatt> ehh, lets not layer everything onto p2p extensions 12:34 < luke-jr> jonasschnelli: XMPP! /s 12:34 -!- d4de [~d4de@41.40.36.81] has joined #bitcoin-core-dev 12:34 < jtimon> why people want to remove zmq? I still don't undesrtand 12:35 < jonasschnelli> sipa: That's why I propose RPC long poll (as a "wallet" feature). :) 12:35 < sipa> jonasschnelli: i'm not sure our wallet should provide that 12:35 < sipa> jonasschnelli: at least not at this stage 12:35 < wumpus> BlueMatt: well if we have a autenticated+encryped P2P protocol, adding private extensions is attractive 12:35 < wumpus> jtimon: I don't want to remove zmq. But I do think we should have one notification mechanism. 12:36 < kanzure> notifications over zeromq would be nice 12:36 < jtimon> wumpus: why not it be zmq? 12:36 < luke-jr> jtimon: I just want to keep ZMQ as an optional feature, not necessary for this stuff 12:36 < wumpus> jtimon: I don't want to amintain an endless pile-up of different external notification mechanisms 12:36 < jonasschnelli> Yes. We don't want that. 12:36 < jtimon> wumpus: ok, why not use ONLY zmq? 12:36 < BlueMatt> wumpus: its quite attractive to shove everything into one monolithic protocol, indeed, but I do think there is a lot of value in splitting things out (though I wouldnt be upset if they were logically different code that just happened to have the same on-wire protocol or so) 12:36 < jcorgan> jtimon: i agree that zmq req/rep and pub/sub could provide the entirety of needed interfaces to bitcoind, but that's an argument lost years ago :) 12:36 < luke-jr> jtimon: then it's no longer optional 12:36 < kanzure> unfortunately the simplest private notification stuff that the rest of the community would understand would be.... web hooks. :( 12:36 < luke-jr> jcorgan: except ZMQ has no protocol compatibility 12:37 < wumpus> websocket would work 12:37 < morcos> wumpus: i think we can move towards deprecating some things that we view as redundant. what we should do now though is decide what would work for a node <-> wallet communication protocol. b/c that is something we defintiely want. 12:37 < jtimon> luke-jr: ok, either we have 1, we remove zmq or we make it non-optional 12:37 < luke-jr> wumpus: websocket doesn't define a protocol 12:37 < wumpus> but just as well as longpoll 12:37 < jonasschnelli> websockets have bad security 12:37 < sipa> wumpus: i think it may be reasonable to say that zmq is for node notifications (which are unauthenticated) and longpoll for wallet notifications 12:37 < luke-jr> jtimon: if we can only have 1, then IMO zmq can go 12:37 < wumpus> morcos: what would you propose to deprecate? 12:37 < jtimon> luke-jr: what do you mean by "protocol compatibility"? 12:38 < luke-jr> jtimon: but I don't think we should limit to 1 12:38 < wumpus> sipa: but what jonasschnelli wants is not wallet notifications 12:38 < luke-jr> jtimon: ZMQ 4.0 can't talk to ZMQ 4.1 12:38 < jtimon> luke-jr: and IMO, if we only want to have one, zmq should stay 12:38 < sipa> wumpus: yes it is 12:38 < wumpus> sipa: he wants the same stuff as is now offered over zmq 12:38 < jonasschnelli> wumpus sipa: What i want is going toward wallet notifications 12:38 < morcos> wumpus: well my argument is to first define the right way to do node <-> wallet and then do that (even if it means adding something) and then we can revisit and see what we have that is extraneous 12:38 < jonasschnelli> But also notifications that could enable a remote GUI 12:38 < sipa> i don't think so; nothing of what is offered over ZMQ is what you need for a remote wallet GUI 12:38 < jonasschnelli> (mempool / peers, etc9 12:38 < jonasschnelli> A Core node GUI on a smartphone.... 12:39 < jonasschnelli> Which can't work over ZMQ 12:39 < sipa> i think we shouldn't discuss that in the current stage 12:39 < jcorgan> of course it *could* 12:39 < jcorgan> but not as it is written today 12:39 < wumpus> jonasschnelli: how would you protect the RPC connection to the smartphone? 12:39 < sipa> i don't want my bitcoin core full node software to be accessible by smartphones on the internet, i think 12:39 < kanzure> jcorgan: how much difference? 12:39 < wumpus> jonasschnelli: the same tunneling tool could route the notifications from zmq 12:40 < jcorgan> zmq is a transport 12:40 < jonasschnelli> wumpus: reverse proxy, SSL auth, mod_security 12:40 < sipa> except the p2p protocol, which is must provide by necessarily 12:40 < morcos> jonasschnelli: don't you think we shoudl concentrate on node <-> wallet now? b/c then different people could build different wallet <-> gui protocols if we wanted. our core responsiblity is the node 12:40 < jcorgan> you have to define a protocol/serialization/encapsulation to run over it 12:40 < luke-jr> jcorgan: except then you need to make sure your ZMQ lib versions match, which is just annoying 12:40 < jcorgan> you can say that about dozens of other libs 12:40 < jonasschnelli> morcos: I'm working on node <-> wallet. That's why I'm working on auth/enc. :) 12:40 < jtimon> jonasschnelli: how can something "not work over ZMQ"? can't you proxy the messages through some other protocol outside of bitcoind once you get the zmq messages? 12:40 < luke-jr> jcorgan: most protocols are compatible across lib versions 12:40 < sipa> jtimon: because authentication 12:40 < jonasschnelli> I don't want to talk SPV over unencrypted channels... 12:41 < kanzure> unencrypted...? 12:41 < jcorgan> anyway, if anyone wants to pursue making changes to the current zmq implementation, we can talk in zurich about it 12:41 < wumpus> wellt he same protocol that offers RPC over the internet needs authentication too 12:41 < jonasschnelli> jtimon: Sure can you. But can a normal user do that? 12:41 < wumpus> you have exactly the same problem there 12:41 < jonasschnelli> I just compared the requirements to setup a remote rtorrent GUI with a possible remote Core node GUI. 12:42 < wumpus> in any case, I'm not strongly against longpoll RPC notifications 12:42 < jtimon> jonasschnelli: well, what was your "normal user" using isntead of zmq? 12:42 < jonasschnelli> And i feal that rpc long polls would result in a easy setup... 12:42 < sipa> morcos, wumpus, jonasschnelli: i agree node <-> wallet is what we should be talking about first, and the apparent need to add private extensions is a sign of a deeper problem: an SPV wallet without trusted full node will be lacking in features 12:42 < kanzure> easiest setup for non-core-developers is going to be web hooks :( 12:42 < kanzure> not rpc 12:43 < d4de> 0MQ supports GSSAPI and Curve auth, unencrypted?! 12:43 < wumpus> and indeed, RPC longpoll can be supported without linking against external dependencies 12:43 < morcos> sipa: good point, but maybe thats a separate problem to solve? 12:43 < wumpus> which is an advantage 12:43 < jonasschnelli> I'm happy to keep the PR alive... I'll also try to work on a intermediate script that could -> ZMQ and does -> RPC 12:43 < sipa> morcos: well if that problem is solved (unsure how...), maybe we don't need private extensions :) 12:43 < sipa> jonasschnelli: you can't... ZMQ doesn't offer wallet-filtered events 12:43 < morcos> yes, but it'll always be superior to some degree to have access to a trusted node 12:44 < instagibbs> sipa, do you mean things like access to mempool, etc? 12:44 < wumpus> at the least we should make a clear division about *what* should be offered over RPC longpoll and what over zmq and what over (theoretic) P2P extensions 12:44 < sipa> instagibbs: yes, and fee estimation 12:44 < morcos> so we shouldn't limit ourselves to things that you can't do without that 12:44 < jonasschnelli> sipa: you still can talk RPC... 12:44 < wumpus> so not everything ends up on all three 12:44 < jonasschnelli> sipa: just the notifications need to be ZMQ to avoid polling.. 12:44 < instagibbs> even if you have access to full node over p2p, that doesn't get you that (maybe that's what you meant) 12:44 < jonasschnelli> sipa: thats why I think adding long poll would not change the security aspect. 12:45 < sipa> jonasschnelli: but you don't want to send a ZMQ event for every transaction over the wire... that would consume as much bandwidth as just the p2p protocol 12:45 < wumpus> adding longpoll to RPC won't change any security aspect 12:45 < luke-jr> more, since ZMQ is ASCII :P 12:45 < sipa> luke-jr: wut? 12:45 < kanzure> rpc thread count exhaustion might change a security aspect 12:45 < wumpus> (except if it encourages opening up your RPC port to the internet) 12:45 < jonasschnelli> sipa: Right. Long poll could then be extended to relevant wallet txes. 12:45 -!- d4de [~d4de@41.40.36.81] has quit [Quit: This computer has gone to sleep] 12:45 < wumpus> kanzure: you need user/pass for that, so only the owner can attack it 12:45 < wumpus> kanzure: I would be against unauthenticated longpolls, that's for sure 12:46 < luke-jr> sipa: at least the main protocol design (it transmits binary data as-is, IIRC) 12:46 < wumpus> kanzure: and if you can authenticate you can do *much* worse things than hold up threads 12:46 < jtimon> sipa: of course you can filter stuff with zmq, you can do anything with zmq, is "network complete" 12:46 < jtimon> you may need more processes 12:46 < sipa> jtimon: that's like saying that x86 asm is better than http, because it can do everything 12:46 < jonasschnelli> Another + for RPC long poll: no dependencies... 12:46 < sipa> jtimon: of course you can filter, but the filter would be too late 12:47 < wumpus> so anyhow: I'd say continue your work jonasschnelli 12:47 < jtimon> sipa: I really don't undesrtand what you claim can't be done with zmq 12:47 < kanzure> my zeromq person just left the channel in a huff ("these people are retarded") hah... 12:47 < jonasschnelli> wumpus: Okay.. I keep the PR warm. 12:47 < luke-jr> again, we ALREADY have longpolling, so I don't see why make a big deal of it 12:47 < jtimon> sipa: you want to filter by a set of addresses or something? 12:47 < jonasschnelli> luke-jr: Yes. It's not a big deal... 12:47 < sipa> jtimon: we don't want ZMQ to be authenticated, so it can't provide wallet-specific information, which means the ZMQ client will need to do the filtering 12:48 < wumpus> people seem to have trouble to get zmq to work in practice, maybe if you can provide some easy examples for RPC longpolling then it will be the more popular way to do notifications soon 12:48 < jonasschnelli> sipa: Agree 12:48 < sipa> jtimon: which is duplicating logic, because that's something the wallet should do, not the gui 12:48 < jonasschnelli> wumpus: Okay. I'll add some examples... good point! 12:48 < wumpus> (I had so much trouble convincing joinmarket to use zmq notifications instead of wacky -notifyX scripts that I just gave u) 12:48 < luke-jr> LP is really simple. just make a normal request and wait ☺ 12:48 < jonasschnelli> wumpus: hah .. yes. Another +1 for RPC long poll. :) 12:48 < wumpus> maybe it's the extra dependency, maybe it's unfamiliarity, I don't know. 12:48 < jonasschnelli> People will call exes on every transaction... 12:48 < jtimon> sipa: oh, you don't want authenticated connections going through zmq processes...I don't undesrtand why, but ok, that seems to be the piece I was missing 12:49 < sipa> jtimon: so yes, of course, you can do anything, if you reimplement the wallet in the client, but that was exactly what we were trying to avoid :) 12:49 < gmaxwell> luke-jr: lots of things time out. 12:49 < jonasschnelli> gmaxwell: time out doesnt matter 12:49 < gmaxwell> this discussion seems ratholed. 12:49 < jonasschnelli> Because you have a queue/state on the server 12:49 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 12:49 < wumpus> timeout doesn't matter with longpoll, you can just re-request 12:49 < jtimon> sipa: no, I would just use some kind of authentication 12:49 < kanzure> we need more user metric survey data about whehter rpc is really easy for folks. i think all the web app developers only know web hooks and HTTP. 12:50 < wumpus> kanzure: moving away from JSON-RPC as the main RPC API is out of the question at this stage 12:50 < wumpus> just too much software and libraries that assume it 12:50 < sipa> are there other topics? 12:50 < kanzure> er i did not mean to unintentionally advocate for moving away from json-rpc 12:51 < wumpus> doesn't seem to be 12:51 < wumpus> #endmeeting 12:51 < lightningbot> Meeting ended Thu May 12 19:51:11 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) 12:51 < lightningbot> Minutes: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-05-12-19.00.html 12:51 < lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-05-12-19.00.txt 12:51 < lightningbot> Log: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-05-12-19.00.log.html 12:52 < jcorgan> do we have any grand plans or even wishlists for zurich? 12:52 < jtimon> kanzure: I did suggest replacing json-rpc with zmq json req/rep, but I knew very few people would consider it seriously 12:52 < sipa> i'd like to talk about p2p encryption, segwit, signature aggregation, ... 12:52 < jonasschnelli> Agree with sipa. We could also form groups to directly work on stuff. 12:53 < sipa> but the overall goal should just be to facillitate improvements that are in the pipeline by meeting in face... whatever those improvements are 12:53 < kanzure> i would like to do braindumps to get things written down 12:53 < jonasschnelli> +1 for writing down things... 12:53 < kanzure> i am good at this 12:53 < jcorgan> suggest doing as much brainstorming/dumping in advance to not use precious F2F time 12:54 < kanzure> would like to draw up list of too-often-unspoken proposals to write down more details about 12:54 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 12:54 < sipa> that'd be great 12:54 < jonasschnelli> I think we should't feel too much pressure. The fact that we are meeting face to face alone is a value. 12:54 < jcorgan> of course 12:54 < sipa> agree 12:54 < jonasschnelli> Even if there is no outcome. It would be a success. 12:55 < kanzure> some feedback from d4de- he recommends that future bitcoin core meetings shoud have someone that more actively guides discussion, e.g. make things more explicit rather than implicit 12:55 < wumpus> jtimon: right - there's no deep reason why zmq req/rep wouldn't work for the main RPC API, but it's .... unfamiliar for people, see my comment above about trying to get people to even use zmq for basic notifications 12:55 < jcorgan> the face time is extremely valuable, just want to make the most of it 12:55 < kanzure> jonasschnelli: gosh that sounds like one of those unreasonable metrics :) (kidding) 12:55 < jonasschnelli> kanzure: It's called positiv thinking. :) 12:55 < wumpus> kanzure: are you volunteering? 12:56 < jcorgan> so if there are things that can be done ahead of time that don't necessarily need the f2f time, it would make sense to get it out of the way beforehand 12:56 < kanzure> wumpus: for writing things down? yes. for brainstorming? sure... for meetings, i'll defer to our friendly neighborhod pointy-hair. 12:56 < jonasschnelli> kanzure: I think wumpus is doing a fantastic job and you easly overmoderate on IRC. 12:56 < gmaxwell> wumpus: execpt the no stable on the wire format, and no reliable delivery... 12:57 < jonasschnelli> jcorgan: Agree. You could start a discussion on the bitcoin-core-dev ML? 12:57 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev 12:57 < kanzure> jonasschnelli: i don't have enough context on many of these topics to correctly guide meetings. so it's not something you should be concerned about. heh. 12:57 -!- mrkent [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 12:57 < wumpus> gmaxwell: I think you can have reliable delivry over zmq, just not in the way we use it 12:57 < jtimon> wumpus: yeah, it seems people forgot about socket-like things 12:57 < wumpus> jtimon: well it's a complete new API to learn 12:58 < wumpus> jtimon: even if you already know normal socket programming, zmq is like a whole new alien landscape 12:58 < jonasschnelli> jtimon: RPC long poll require no "real" dependencies, you just aim a http request and wait until you get response. 12:58 < jcorgan> wumpus: jtimon: people don't seem to grok message based vs. function call based 12:58 < kanzure> jonasschnelli: that can cause an unintentional DoS against yourself 12:58 < jonasschnelli> kanzure: explain? 12:58 < kanzure> rpc thread exhaustion, as mentioned above 12:58 < jtimon> wumpus: I hadn't used sockets in ages and it was very easy to me (but as said I just use xreq/xrep, push/pull and pub/sub, no dealers or other stuff) 12:58 < jcorgan> jonasschnelli: i'll make a suggestion on the ML then 12:59 < jonasschnelli> jcorgan:+1 12:59 < jonasschnelli> kanzure: no.. you only run the poll http request on a single thread... 12:59 < jonasschnelli> Only one request in parallel. 12:59 < kanzure> jonasschnelli: rpcthreads 12:59 < jonasschnelli> But thats the serverside. 12:59 < kanzure> yes.. that's still DoSing yourself. 13:00 < kanzure> you're not enforcing one request in parallel 13:00 < jtimon> wumpus: but apparently nanomsg simplifies things further removing the context and things like that 13:00 < wumpus> in principle you don't even need a rpc thread open for a long-poller, if you'd use the event based http, but that's a topic for the far future 13:00 < jonasschnelli> kanzure: sure. But that would not be different to the "normal" rpc calls? 13:00 < kanzure> jtimon: in addition to nanomsg, look at nq 13:00 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 276 seconds] 13:00 < kanzure> jonasschnelli: well, not many of them are meant to be open for a while 13:01 < jtimon> kanzure: nq? do you have a link? 13:01 < jonasschnelli> Sure. But you control them. Its secured behind auth. 13:01 < jonasschnelli> kanzure: No strangers will connect to your RPC. 13:01 < kanzure> jtimon: actually no :) i think this means i owe you BTC... hah. 13:01 < wumpus> and if you need multiple long-pollers you can just increase the number of rpc threads anyhow, it's not like we need to support 1000 people in a chat 13:01 < wumpus> no need to overdesign/overcomplicate 13:01 < jonasschnelli> Indeed. :) 13:02 < jtimon> jonasschnelli: you can have an http server with auth that connects "locally" (intranet) to stuff via zmq 13:02 < kanzure> manual thread management, got it.. 13:02 < luke-jr> we're moving the dev meetings to the p2p network and using longpolling with bitcoin-cli to read the chat? :P 13:02 < jtimon> jonasschnelli: but yeah, I guess that's one extra step you would be saving 13:02 < jonasschnelli> jtimon: Yes. Sure. But why the roundtrip when you can have RPC long poll? 13:02 < kanzure> we don't mention in our docs "don't connect to your bitcoin node from a thousand workers that listen for notifications" 13:02 < jonasschnelli> jtimon: you don't need the ZMQ dependencies if you want to serve a push not. over http 13:02 < wumpus> kanzure: some things are common sense, but feel free to add it explicitly 13:03 < wumpus> kanzure: and for that, zmq is indeed more suited as it has one-to-many broadcast 13:04 < jtimon> jonasschnelli: of course, strictly speaking "you don't need anything" to do anything, it's the "you can't do X using zmq" that nerves me 13:04 < jonasschnelli> I think ZMQ is useful.... 13:04 < gmaxwell> jtimon: being excessively pedantic is not productive. 13:05 < jonasschnelli> Its just pain in the ass to get a notification to a remote end. 13:06 < jtimon> gmaxwell: sure, I was just misunderstanding some of the "you can't do this with zmq" claims, they were really just "you also need X" or similar things 13:06 < jtimon> mistery solved 13:07 * jonasschnelli needs rest and waves goodby... 13:08 < jtimon> and I'm sorry for being so biased in favor of zmq whenever it's an option for anything, I guess it may equilibrate some other biases against it. 13:10 -!- cryptapus [~cyptapus@unaffiliated/cryptapus] has quit [Ping timeout: 276 seconds] 13:14 -!- BashCo_ [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 13:15 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 13:17 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Ping timeout: 276 seconds] 13:18 < jcorgan> one suggestion--we seem to be going back and forth between different use cases and different means to accomplish them 13:19 < jcorgan> i'd rather clarify what all the use cases are first, then see what technology best suits them, then see how that maps to what exists and what doesn't in the client 13:22 < jcorgan> *best suits each of them 13:26 < jtimon> jcorgan: that sounds very reasonable 13:26 < jtimon> analysis first, arch/design decisions later 13:34 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has quit [Ping timeout: 252 seconds] 13:46 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev 13:47 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Remote host closed the connection] 13:47 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev 13:52 -!- cryptapus_afk is now known as cryptapus 13:52 -!- cryptapus is now known as cryptapus_afk 13:52 -!- cryptapus_afk is now known as cryptapus 14:02 -!- jcorgan [~jcorgan@unaffiliated/jcorgan] has quit [Quit: leaving] 14:12 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Ping timeout: 276 seconds] 14:12 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev 14:17 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Ping timeout: 246 seconds] 14:19 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev 14:37 -!- d4de [~d4de@41.40.36.81] has joined #bitcoin-core-dev 14:44 -!- d4de [~d4de@41.40.36.81] has quit [Quit: This computer has gone to sleep] 14:44 -!- d4de [~d4de@41.40.36.81] has joined #bitcoin-core-dev 14:46 -!- zooko [~user@c-73-229-199-227.hsd1.co.comcast.net] has joined #bitcoin-core-dev 14:52 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 14:57 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 15:00 -!- AaronvanW [~ewout@unaffiliated/aaronvanw] has quit [Remote host closed the connection] 15:03 -!- mrkent [~textual@unaffiliated/mrkent] has quit [] 15:03 -!- cryptapus is now known as cryptapus_afk 15:04 -!- mrkent [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 15:12 -!- gabridome [~gabridome@net-93-145-200-45.cust.vodafonedsl.it] has joined #bitcoin-core-dev 15:20 -!- mrkent [~textual@unaffiliated/mrkent] has quit [] 15:24 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 15:28 -!- mrkent [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 15:35 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has quit [] 15:38 -!- gabridome [~gabridome@net-93-145-200-45.cust.vodafonedsl.it] has quit [Quit: gabridome] 15:39 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 15:40 -!- BashCo_ [~BashCo@unaffiliated/bashco] has quit [Ping timeout: 250 seconds] 15:54 < BlueMatt> ugh, master is segfaulting trying to call rpc 15:55 < sipa> BlueMatt: elaborate? 15:55 < BlueMatt> gdb'ing 15:55 < sipa> thanks 15:55 < BlueMatt> but all i did was start and then try to call rpc :( 15:57 < BlueMatt> oh, false alarm...custom change blew up 15:57 < BlueMatt> sorry for the noise :/ 15:57 < BlueMatt> fucking 1-line change segfaults :( 15:58 < sipa> we need a programming language with built-in reed-selomon code 15:58 < sipa> so that the compiler can correct 1-line errors :p 15:58 < BlueMatt> heh 15:58 < gmaxwell> sipa: and updating the code means you need to read all of it? 15:59 < sipa> gmaxwell: you mean you don't already do that? 15:59 -!- murch [~murch@p4FE3B1CE.dip0.t-ipconnect.de] has quit [Quit: Leaving.] 16:05 -!- d4de [~d4de@41.40.36.81] has quit [Quit: This computer has gone to sleep] 16:07 -!- PRab [~chatzilla@c-68-62-95-247.hsd1.mi.comcast.net] has joined #bitcoin-core-dev 16:13 -!- amiller [~socrates1@unaffiliated/socrates1024] has quit [Ping timeout: 244 seconds] 16:18 -!- Guest13955 [~socrates1@li175-104.members.linode.com] has joined #bitcoin-core-dev 16:22 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)] 16:23 -!- aj [aj@cerulean.erisian.com.au] has quit [Ping timeout: 246 seconds] 16:25 -!- wangchun [~wangchun@li414-193.members.linode.com] has quit [Ping timeout: 265 seconds] 16:25 -!- wangchun [~wangchun@li414-193.members.linode.com] has joined #bitcoin-core-dev 16:30 -!- aj [~aj@cerulean.erisian.com.au] has joined #bitcoin-core-dev 16:45 -!- zooko [~user@c-73-229-199-227.hsd1.co.comcast.net] has quit [Ping timeout: 276 seconds] 16:50 -!- TomMc [~tom@unaffiliated/tommc] has quit [Ping timeout: 260 seconds] 16:55 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 16:56 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving] 17:00 -!- fengling [~fengling@111.198.29.53] has quit [Ping timeout: 240 seconds] 17:08 -!- alpalp [~allen@104-54-235-28.lightspeed.austtx.sbcglobal.net] has joined #bitcoin-core-dev 17:09 -!- alpalp [~allen@104-54-235-28.lightspeed.austtx.sbcglobal.net] has quit [Changing host] 17:09 -!- alpalp [~allen@unaffiliated/alpalp] has joined #bitcoin-core-dev 17:15 < sipa> $ ./walletbackup.py 17:15 < sipa> INFO: Restoring using dumped wallet 17:15 < sipa> Unexpected exception caught during testing: BrokenPipeError(32, 'Broken pipe') 17:15 < sipa> Stopping nodes 17:15 < sipa> WARN: Unable to stop node: CannotSendRequest('Request-sent',) 17:15 < gmaxwell> sipa: patrick got to you? 17:16 < gmaxwell> the question is-- why doesn't travis see it as failing? 17:18 < sipa> gmaxwell, wumpus, jonasschnelli: 17:19 < gmaxwell> sipa, sipa, sipa 17:20 < sipa> gmaxwell: are you wumpus and jonasschnelli? 17:21 < sipa> i wanted to paste my rorx results, but they seem to have disappeared into my terminal's forgotten history 17:21 < gmaxwell> No, are you? 17:21 < sipa> both rorxes were similar, and fastest 17:21 < gmaxwell> well, see.. I saved them seconds of disappointment by wisely preempting your summons. 17:24 < gmaxwell> sipa: did you see the usenix paper I linked to you some indeterminable number of days ago? 17:24 < sipa> i skimmed it, not enough to actually find the xor trick you were referring to 17:25 < gmaxwell> are you familar with cuckoo hashing? 17:25 -!- d4de [~d4de@41.43.204.131] has joined #bitcoin-core-dev 17:26 < sipa> yes 17:27 < gmaxwell> So how do you support cuckoo hashing if the whole value can't be stored in the table? -- normally you rehash an entry to find its alternative location when you evict it. 17:29 < gmaxwell> The make the primary location H(full value)->to offset and the secondary location H(short value) ^ location. So you only need the current location and the short value to swap an entry between slots. 17:29 < gmaxwell> (and you don't need to track which one it was in, since the same xor relates them) 17:32 < sipa> gmaxwell, wumpus, jonasschnelli: 17:32 < sipa> SHA256_avx,175,0.00575656,0.0058105,0.00575941 17:32 < sipa> SHA256_basic,119,0.0088715,0.00893307,0.00887388 17:32 < sipa> SHA256_rorx,223,0.00482899,0.004861,0.00483046 17:32 < sipa> SHA256_rorx_x8ms,223,0.00475737,0.0047915,0.00476165 17:32 < sipa> SHA256_sse4,175,0.00574069,0.00577295,0.00574323 17:33 < sipa> i7-4800MQ, fixed at 2.6 GHz 17:34 -!- alpalp [~allen@unaffiliated/alpalp] has quit [Quit: Konversation terminated!] 17:34 < phantomcircuit> is the first column cycle count? 17:34 -!- alpalp [~allen@104-54-235-28.lightspeed.austtx.sbcglobal.net] has joined #bitcoin-core-dev 17:34 -!- alpalp [~allen@104-54-235-28.lightspeed.austtx.sbcglobal.net] has quit [Changing host] 17:34 -!- alpalp [~allen@unaffiliated/alpalp] has joined #bitcoin-core-dev 17:34 < sipa> iteration count 17:34 < gmaxwell> phantomcircuit: it's number of times the freaky bench innerloop ran. 17:34 < phantomcircuit> ah 17:34 < sipa> how many tests were done 17:35 < sipa> each doing 1 MB of data 17:35 < phantomcircuit> ok 17:35 < phantomcircuit> neat 17:35 < gmaxwell> really for our usage running with 32 and 64 bytes of data is much more interesting. 17:36 < gmaxwell> might be useful to see if that changes the numbers at all... perhaps gives AVX a purpose for existing? :) 17:36 < sipa> sure, but it's just a proxy for the number of sha256 compression function runs 17:36 -!- flybyguy [d8430576@gateway/web/freenode/ip.216.67.5.118] has joined #bitcoin-core-dev 17:37 < gmaxwell> ah duh right, it doesn't have a finialization, so it's not going to change. 17:37 -!- flybyguy [d8430576@gateway/web/freenode/ip.216.67.5.118] has left #bitcoin-core-dev [] 17:37 < sipa> note, it's a mobile CPU; if i don't fix the clock speed, cpufreq increases my cpu speed somewhere during the rorx_x8m run 17:37 < phantomcircuit> gmaxwell, loading into the right registers takes some amount of work 17:37 < sipa> making it look wildly better than rorx 17:38 < phantomcircuit> sipa, test on a build box? 17:38 < phantomcircuit> gmaxwell, any idea if those have turbo or not? 17:38 < sipa> turbo is easy to disable 17:41 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-lrxwwskpigyvqowb] has quit [Quit: Connection closed for inactivity] 17:46 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev 17:47 -!- d4de [~d4de@41.43.204.131] has quit [Quit: Leaving] 17:48 < sipa> on our 56-core machine: 17:48 < sipa> SHA256_avx,255,0.00397131,0.00401497,0.00397456 17:48 < sipa> SHA256_basic,175,0.00599988,0.00604987,0.00600181 17:48 < sipa> SHA256_rorx,319,0.00334556,0.003407,0.00334662 17:48 < sipa> SHA256_rorx_x8ms,319,0.00328553,0.00332999,0.00328678 17:48 < sipa> SHA256_sse4,255,0.00395919,0.00401807,0.00396108 17:48 < sipa> gmaxwell: what cpu is it? 17:52 < sipa> i'm surprised that our C code is only 45% slower than intel's optimized asm code 17:53 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Quit: laurentmt] 17:56 < gmaxwell> the 56-core are broadwell-ep 17:56 < gmaxwell> but 'only' at 2.2GHz. 17:58 < phantomcircuit> sipa, none of those are parallel calculation right? 17:59 < sipa> indeed, all single threaded 18:00 -!- Guest13955 [~socrates1@li175-104.members.linode.com] has quit [Changing host] 18:00 -!- Guest13955 [~socrates1@unaffiliated/socrates1024] has joined #bitcoin-core-dev 18:00 -!- Guest13955 is now known as amiller 18:00 < gmaxwell> sipa: sha2 that computes four at once is a considerable additional speedup, but harder to use without more software changes. 18:01 < sipa> very doable inside merkle trees, though 18:02 -!- mrkent_ [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 18:02 < gmaxwell> Yes. though thats pretty much the only place where it's very doable. 18:02 < sipa> also the place where it probably matters most 18:03 < gmaxwell> I'm sure if you want to write the merkle tree function wumpus will hapily do the work to give you a sha2x4 to call. :) 18:04 -!- mrkent [~textual@unaffiliated/mrkent] has quit [Ping timeout: 260 seconds] 18:05 -!- dermoth_ [~thomas@dsl-66-36-143-241.mtl.aei.ca] has quit [Read error: Connection reset by peer] 18:06 -!- dermoth_ [~thomas@dsl-66-36-143-241.mtl.aei.ca] has joined #bitcoin-core-dev 18:12 < gmaxwell> my vague recollection was "the asm was 2x faster than the C, and the 4-way was 3x faster than the C" I dunno how that generalizes with the rorx. 18:12 -!- jtimon [~quassel@65.28.134.37.dynamic.jazztel.es] has quit [Ping timeout: 252 seconds] 18:28 -!- fengling [~fengling@111.198.29.53] has joined #bitcoin-core-dev 18:34 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has quit [Remote host closed the connection] 18:35 -!- Giszmo [~leo@190.46.14.122] has joined #bitcoin-core-dev 18:40 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection] 18:41 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev 19:02 -!- Giszmo [~leo@190.46.14.122] has quit [Ping timeout: 240 seconds] 19:13 -!- helo [~helo@unaffiliated/helo] has quit [Ping timeout: 264 seconds] 19:14 -!- helo [~helo@unaffiliated/helo] has joined #bitcoin-core-dev 19:22 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has joined #bitcoin-core-dev 19:36 -!- alpalp [~allen@unaffiliated/alpalp] has quit [Ping timeout: 246 seconds] 19:43 -!- CubicEarth [~cubiceart@c-73-68-232-79.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 19:44 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 246 seconds] 19:52 -!- achow101 [~achow101@pool-96-227-114-115.phlapa.fios.verizon.net] has quit [Read error: Connection reset by peer] 19:53 < GitHub130> [bitcoin] sipa opened pull request #8051: Fix walletbackup.py failure (master...fixwalletbackup) https://github.com/bitcoin/bitcoin/pull/8051 19:57 -!- dermoth__ [~thomas@dsl-66-36-159-8.mtl.aei.ca] has joined #bitcoin-core-dev 19:59 -!- dermoth_ [~thomas@dsl-66-36-143-241.mtl.aei.ca] has quit [Ping timeout: 246 seconds] 20:02 < gmaxwell> sipa: considering your PR comments might it be a bit strong to call that 'fix'? rather than 'mysteriously stir'? :) 20:04 < sipa> gmaxwell: well, it's reproducible :) 20:04 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Read error: Connection reset by peer] 20:04 < sipa> maybe i should call it "seemingly fix" 20:05 < sipa> done 20:05 -!- mrkent_ [~textual@unaffiliated/mrkent] has quit [] 20:05 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has joined #bitcoin-core-dev 20:05 < gmaxwell> presumably it fails for you because your computer is fast and travis isn't. 20:06 -!- lecusemble [~lecusembl@104.233.76.133] has quit [Ping timeout: 276 seconds] 20:06 < sipa> hmm, i started off by adding sleeps and that didn't help 20:06 < sipa> but let me try again, by adding a sleep exactly there 20:19 < sipa> gmaxwell: the sync blocks there causes a sleep of 1-2 seconds 20:19 < sipa> gmaxwell: an explicit sleep of 60s does not fix it 20:23 -!- lecusemble [~lecusembl@104.233.76.133] has joined #bitcoin-core-dev 20:24 < gmaxwell> uh how does that make any sense? 20:24 < sipa> sense, it makes none 21:52 -!- lecusemble [~lecusembl@104.233.76.133] has quit [Ping timeout: 265 seconds] 22:02 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Ping timeout: 276 seconds] 22:05 -!- lecusemble [~lecusembl@104.233.76.133] has joined #bitcoin-core-dev 22:13 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Ping timeout: 260 seconds] 22:32 -!- gabridome [~gabridome@net-93-145-200-45.cust.vodafonedsl.it] has joined #bitcoin-core-dev 22:38 -!- Arnavion [arnavion@unaffiliated/arnavion] has quit [Quit: Arnavion] 22:38 -!- Arnavion [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev 22:48 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev 22:52 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has quit [Ping timeout: 250 seconds] 22:53 -!- Arnavion [arnavion@unaffiliated/arnavion] has quit [Quit: Arnavion] 22:54 -!- Arnavion [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev 23:12 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-uwzkyvgysgyiugfo] has joined #bitcoin-core-dev 23:15 -!- Arnavion [arnavion@unaffiliated/arnavion] has quit [Quit: Arnavion] 23:15 -!- Arnavion [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev 23:48 -!- gabridome [~gabridome@net-93-145-200-45.cust.vodafonedsl.it] has quit [Quit: gabridome] 23:53 < jonasschnelli> sipa: I read something in the logs: is walletbackup.py fixed? 23:53 * jonasschnelli reading https://github.com/bitcoin/bitcoin/pull/8051#issuecomment-218943822 23:53 < sipa> jonasschnelli: for me it is 23:54 -!- gabridome [~gabridome@net-93-145-200-45.cust.vodafonedsl.it] has joined #bitcoin-core-dev