--- Log opened Sat Aug 25 00:00:33 2018 --- Day changed Sat Aug 25 2018 00:00 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has quit [Remote host closed the connection] 00:00 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has joined #bitcoin-core-dev 00:18 -!- Lauda [~quassel@unaffiliated/lauda] has joined #bitcoin-core-dev 00:29 -!- vexbuy [~vexbuy@89.39.107.192] has joined #bitcoin-core-dev 00:37 -!- murrayn [~dafuq@unaffiliated/murrayn] has quit [Read error: Connection reset by peer] 00:39 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev 00:39 -!- wxss [~user@5.187.21.78] has joined #bitcoin-core-dev 00:43 -!- promag [~promag@83.223.235.81] has quit [Ping timeout: 244 seconds] 00:53 -!- Rootsudo [~textual@180.190.116.243] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 00:54 -!- Rootsudo [~textual@180.190.116.243] has joined #bitcoin-core-dev 00:55 -!- Rootsudo [~textual@180.190.116.243] has quit [Client Quit] 00:55 -!- Rootsudo [~textual@180.190.116.243] has joined #bitcoin-core-dev 00:55 -!- Rootsudo [~textual@180.190.116.243] has quit [Client Quit] 00:57 -!- Rootsudo [~textual@180.190.116.243] has joined #bitcoin-core-dev 00:57 -!- Rootsudo [~textual@180.190.116.243] has quit [Client Quit] 01:04 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has joined #bitcoin-core-dev 01:05 -!- vexbuy [~vexbuy@89.39.107.192] has quit [Remote host closed the connection] 01:07 -!- vexbuy [~vexbuy@89.39.107.192] has joined #bitcoin-core-dev 01:08 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has joined #bitcoin-core-dev 01:11 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has quit [Quit: Leaving.] 01:12 -!- vexbuy [~vexbuy@89.39.107.192] has quit [Ping timeout: 268 seconds] 01:34 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 01:35 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 01:40 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has quit [Remote host closed the connection] 01:42 -!- vexbuy [~vexbuy@89.39.107.192] has joined #bitcoin-core-dev 01:42 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has joined #bitcoin-core-dev 02:08 -!- csknk [~csknk@unaffiliated/csknk] has joined #bitcoin-core-dev 02:22 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev 02:22 -!- promag [~promag@83.223.235.81] has quit [Remote host closed the connection] 02:37 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has quit [Ping timeout: 252 seconds] 03:03 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 03:03 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection] 03:06 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 03:22 -!- profmac [~ProfMac@2001:470:1f0f:226:21c7:9b4e:db96:accb] has quit [Ping timeout: 264 seconds] 03:30 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev 03:35 -!- promag [~promag@83.223.235.81] has quit [Ping timeout: 268 seconds] 03:45 -!- cdecker [~cdecker@mail.snyke.net] has quit [Quit: ZNC - http://znc.in] 04:27 -!- vexbuy [~vexbuy@89.39.107.192] has quit [Remote host closed the connection] 04:34 -!- goatpig [56eece80@gateway/web/freenode/ip.86.238.206.128] has quit [Ping timeout: 252 seconds] 04:40 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 04:42 -!- promag [~promag@83.223.234.98] has joined #bitcoin-core-dev 04:42 -!- IGHOR [~quassel@93.178.216.72] has quit [Read error: No route to host] 04:43 -!- IGHOR [~quassel@93.178.216.72] has joined #bitcoin-core-dev 04:46 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [] 04:46 -!- promag [~promag@83.223.234.98] has quit [Ping timeout: 252 seconds] 05:04 -!- promag [~promag@83.223.234.98] has joined #bitcoin-core-dev 05:11 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 05:36 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 05:37 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 05:41 -!- as1nc [~as1nc@ax213-1-82-66-157-194.fbx.proxad.net] has joined #bitcoin-core-dev 05:41 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 05:42 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 05:46 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Ping timeout: 252 seconds] 05:47 -!- promag [~promag@83.223.234.98] has quit [Remote host closed the connection] 05:48 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 05:56 -!- esotericnonsense [~esotericn@unaffiliated/esotericnonsense] has quit [Ping timeout: 268 seconds] 06:25 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 06:39 -!- promag [~promag@83.223.234.98] has joined #bitcoin-core-dev 06:40 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 272 seconds] 06:43 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Read error: Connection reset by peer] 06:44 -!- promag [~promag@83.223.234.98] has quit [Ping timeout: 268 seconds] 06:44 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev 06:55 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 06:56 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection] 06:58 -!- owowo [~ovovo@unaffiliated/ovovo] has quit [Ping timeout: 264 seconds] 07:06 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 07:08 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 272 seconds] 07:11 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 244 seconds] 07:13 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 07:23 -!- owowo [~ovovo@unaffiliated/ovovo] has joined #bitcoin-core-dev 07:30 -!- promag [~promag@83.223.234.98] has joined #bitcoin-core-dev 07:43 -!- promag [~promag@83.223.234.98] has quit [Remote host closed the connection] 08:03 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 08:04 -!- vexbuy [~vexbuy@89.39.107.194] has joined #bitcoin-core-dev 08:04 -!- bsm117532 [~mcelrath@c-24-61-184-150.hsd1.ma.comcast.net] has joined #bitcoin-core-dev 08:07 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 08:08 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 268 seconds] 08:12 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 252 seconds] 08:17 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 08:20 -!- vexbuy [~vexbuy@89.39.107.194] has quit [Ping timeout: 244 seconds] 08:22 -!- ossifrage [~ossifrage@unaffiliated/ossifrage] has quit [Remote host closed the connection] 08:23 -!- ossifrage [~ossifrage@unaffiliated/ossifrage] has joined #bitcoin-core-dev 08:40 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 09:16 -!- drexl [~drexl@cpc130676-camd16-2-0-cust445.know.cable.virginm.net] has joined #bitcoin-core-dev 09:33 -!- Zenton [~user@54.104.135.37.dynamic.jazztel.es] has quit [Ping timeout: 244 seconds] 09:38 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 09:39 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 09:41 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has joined #bitcoin-core-dev 09:49 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has quit [Ping timeout: 276 seconds] 09:55 -!- CubicEarth [~CubicEart@c-73-181-185-197.hsd1.wa.comcast.net] has quit [Ping timeout: 260 seconds] 10:02 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 10:13 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has joined #bitcoin-core-dev 10:15 -!- spinza [~spin@155.93.246.187] has quit [Ping timeout: 244 seconds] 10:26 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev 10:35 -!- Taifta [~Taifta@141.226.151.137] has joined #bitcoin-core-dev 10:45 -!- jhfrontz [~Adium@cpe-184-57-118-36.columbus.res.rr.com] has quit [Quit: Leaving.] 10:48 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has joined #bitcoin-core-dev 10:49 -!- jhfrontz [~Adium@cpe-184-57-118-36.columbus.res.rr.com] has joined #bitcoin-core-dev 11:08 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds] 11:20 -!- drexl [~drexl@cpc130676-camd16-2-0-cust445.know.cable.virginm.net] has quit [Read error: Connection reset by peer] 11:29 -!- csknk [~csknk@unaffiliated/csknk] has quit [Quit: leaving] 11:51 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has quit [Remote host closed the connection] 11:58 -!- davex__ [~user@67-3-156-44.omah.qwest.net] has joined #bitcoin-core-dev 12:03 -!- hebasto [~hebasto@195.60.70.234] has joined #bitcoin-core-dev 12:07 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has joined #bitcoin-core-dev 12:10 -!- ken2812221 [~ken281222@110.50.144.85] has quit [Ping timeout: 252 seconds] 12:21 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 12:22 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 12:32 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has quit [Remote host closed the connection] 12:32 -!- wumpus [~wumpus@pdpc/supporter/professional/wumpus] has joined #bitcoin-core-dev 13:39 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Ping timeout: 244 seconds] 13:39 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 13:39 < jamesob> re: memory usage increase: preliminary bisections are in and MarcoFalke and I are betting it's the leveldb changes. https://i.imgur.com/8aXRzwe.png 13:41 -!- intcat [~zshlyk@gateway/tor-sasl/intcat] has quit [Ping timeout: 250 seconds] 13:42 < gmaxwell> jamesob: wait. how are we measuring memory usage in that benchmark? 13:42 < jamesob> gmaxwell: time -f %M (ie resident set size) 13:49 < jamesob> https://github.com/chaincodelabs/bitcoinperf/blob/master/runner/main.py#L659-L668 13:53 < sipa> so every mmap causes 150 kB RSS increase? :s 13:54 < gmaxwell> no because we're not actually using the new maximum. 13:54 < gmaxwell> (I mean not using all of) 13:54 < sipa> aha. 13:54 < gmaxwell> So even MOAR. 13:55 < gmaxwell> which is suspect 13:55 < sipa> mmap by default will prefetch mapped pages into memory 13:55 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has quit [Ping timeout: 272 seconds] 13:55 < sipa> by default, 31 4kB blocks 13:55 < gmaxwell> oh interesting. but those pages are clean they'll just get evicted, they really shouldn't be counted bt RSS :( 13:57 < gmaxwell> so MADV_RANDOM 13:57 < gmaxwell> ... 13:57 < sipa> madvice(MADV_DONTNEED) will disable the reading entirely 13:58 < sipa> which may be useful to diagnose the issue 13:58 < jamesob> (for those following along at home: https://github.com/bitcoin/bitcoin/pull/13925/files) 13:58 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection] 13:59 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 13:59 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 13:59 < gmaxwell> Also setting the maximum maps really low, like..2 might be interesting. 14:02 < gmaxwell> but if this is the problem, MADV_RANDOM is probably the fix to the extent that its an actual problem at all. 14:02 < gmaxwell> Though we should do a reindex benchmark to make sure MADV_RANDOM doesn't hurt performance. 14:03 < wumpus> PSA: if after the latest merge you get a linker error "/usr/local/include/boost/smart_ptr/shared_ptr.hpp:728: undefined reference to `translationInterface", you need to do a 'make clean' and re-do the make and it will work 14:03 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 268 seconds] 14:04 < luke-jr> wumpus: is there a reason the gitian linux yml has g++-riscv64-linux-gnu as a dep? seems to pull in GCC 7 when we're using GCC 8 now? 14:04 < luke-jr> wumpus: if `make clean` ever fixes something, that means there's a bug in the build system :/ 14:06 < sipa> gmaxwell, jamesob: LMDB uses MADV_RANDOM it seems 14:06 < sipa> (though its design is different, i don't know their access patterns) 14:06 < wumpus> luke-jr: yes, it must be missing some changes in dependency detection between source and header files (another one is if you change something in univalue, it won't detect it) 14:06 < sipa> jamesob: try this: 14:06 < sipa> void* base = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0); 14:06 < sipa> if (base != MAP_FAILED) { 14:06 < sipa> + madvise(base, size, MADV_DONTNEED); 14:07 < sipa> in src/leveldb/util/env_posix.cc 14:07 < sipa> (and/or MADV_RANDOM) 14:07 < wumpus> luke-jr: it pulls in both gcc 7 and 8, I think that's necessary due to some strangeness with the packages (some symlink will only exist when g++-riscv64-linux-gnu is also installed) 14:07 < luke-jr> DONTNEED sounds wrong? 14:07 < luke-jr> wumpus: ah, weird 14:08 < wumpus> luke-jr: you might be able to get around it, but I noticed and tried as well and ran into a dead end 14:08 < sipa> luke-jr: to diagnoze 14:08 < sipa> luke-jr: it would be interesting to see what the effect on RSS is with DONTNEED, to have an idea to what extent our memory usage is due to mmap caching 14:09 < jamesob> sipa: giving it a shot now 14:09 < wumpus> luke-jr: at least it's not MADV_HWPOISON! 14:10 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 14:10 -!- intcat [~zshlyk@gateway/tor-sasl/intcat] has joined #bitcoin-core-dev 14:10 -!- plankers [~plank@200.7.98.14] has joined #bitcoin-core-dev 14:10 < luke-jr> wumpus: lol 14:11 < gmaxwell> It's plausable to me that MADV_RANDOM will help performance or at least be neutral. 14:11 < wumpus> yes, to me too 14:11 < wumpus> our access pattern is more or less random 14:12 < gmaxwell> I don't recall now, though I know I researched this before... does leveldb's bisection interpolate assuming keys are uniform and that their values are uniformly sized or does it plain bisect? 14:12 < wumpus> (except in the rare times it's iterating over the whole utxo set in order, like when computing statistics) 14:13 < gmaxwell> esp in the case of plain bisection, prefetching is a bad behavior. 14:13 < wumpus> I don't know 14:13 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 14:14 -!- vexbuy [~vexbuy@217.23.3.171] has joined #bitcoin-core-dev 14:15 < sipa> gmaxwell: there is an index at the end of each ldb file 14:15 < sipa> i assume it bisects in that index in a naive way, but i'm not sure 14:21 < jamesob> sipa: bench is running; we'll know how your change works in about six hours 14:22 < sipa> jamesob: awesome 14:22 < luke-jr> hm, I think I will regret relatime when I try to prune gitian caches 14:53 -!- meshcollider_ [uid246294@gateway/web/irccloud.com/x-baedxxadrwiqngep] has joined #bitcoin-core-dev 14:54 -!- Amuza [~amuza@212.230.156.111] has joined #bitcoin-core-dev 14:55 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 14:56 -!- vexbuy [~vexbuy@217.23.3.171] has quit [Read error: Connection reset by peer] 15:02 -!- jhfrontz [~Adium@cpe-184-57-118-36.columbus.res.rr.com] has quit [Read error: Connection reset by peer] 15:08 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 15:09 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 15:11 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 15:17 -!- plankers [~plank@200.7.98.14] has quit [Ping timeout: 264 seconds] 15:29 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 15:32 -!- plankers [~plank@38.87.81.82] has joined #bitcoin-core-dev 15:32 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 15:33 -!- Amuza [~amuza@212.230.156.111] has quit [Ping timeout: 252 seconds] 15:48 -!- esotericnonsense [~esotericn@unaffiliated/esotericnonsense] has joined #bitcoin-core-dev 15:57 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...] 16:07 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has quit [Ping timeout: 250 seconds] 16:11 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has joined #bitcoin-core-dev 16:18 -!- dcousens [~dcousens@110.140.174.10] has quit [Ping timeout: 255 seconds] 16:23 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 16:24 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 16:24 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev 16:31 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 268 seconds] 16:40 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has quit [Ping timeout: 276 seconds] 16:41 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has joined #bitcoin-core-dev 17:03 -!- meshcollider_ [uid246294@gateway/web/irccloud.com/x-baedxxadrwiqngep] has quit [Quit: Connection closed for inactivity] 17:09 -!- plankers [~plank@38.87.81.82] has quit [Ping timeout: 252 seconds] 17:15 < gmaxwell> sipa: ah that might have been why I didn't bother trying to make it interpolate-- the seeks are in an index. 17:22 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has quit [Ping timeout: 252 seconds] 17:25 -!- plankers [~plank@200.7.98.14] has joined #bitcoin-core-dev 17:29 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has joined #bitcoin-core-dev 17:30 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has quit [Remote host closed the connection] 17:46 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 17:53 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds] 18:31 -!- plankers [~plank@200.7.98.14] has quit [Ping timeout: 272 seconds] 19:34 -!- Jmabsd [~jmabsd@1.186.68.250] has joined #bitcoin-core-dev 19:38 < jamesob> sipa gmaxwell: mem usage increase is for sure the leveldb change (https://i.imgur.com/yMHJI37.png) and sipa's change didn't change the elevated usage 19:39 < sipa> jamesob: so it's still elevated with my patch? 19:39 < sipa> or it's not elevated anymore? 19:40 < jamesob> still elevated :( 19:41 < gmaxwell> might want to make a microbenchmar to figure out if its mmap behavior or something else in leveldb. 19:42 < gmaxwell> also because it'll be easier to expirement (e.g. see if MAP_NORESERVE does anything) when we don't have to wait 5 hours. 19:42 < gmaxwell> just a loop to mmap all the files in a directory then sleep 30 seconds should test the earlier hypothesis. 19:42 -!- Krellan [~Krellan@71.136.136.149] has joined #bitcoin-core-dev 19:43 < jamesob> gmaxwell: yup, agree 19:43 < jamesob> will write if I have time tomorrow 19:50 -!- murrayn [~dafuq@unaffiliated/murrayn] has joined #bitcoin-core-dev 20:12 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 20:13 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 20:16 -!- phwalkr [~phwalkr@2001:1284:f01c:5d1e:d089:d43e:883d:be7a] has joined #bitcoin-core-dev 20:20 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 20:22 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 20:28 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 20:28 -!- Rootsudo [~textual@180.190.116.181] has joined #bitcoin-core-dev 20:30 -!- plankers [~plank@200.7.98.14] has joined #bitcoin-core-dev 20:38 -!- plankers [~plank@200.7.98.14] has quit [Read error: No route to host] 20:46 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 20:47 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 20:51 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Ping timeout: 252 seconds] 20:53 < ossifrage> It seems like MADV_RANDOM is useful if there is less locality to the reads then the size of the prefetch window, but I was under the impression that the prefetch is triggered by the pagefault and not by the mmap() so you actually have to do reads to get the extra pages 21:06 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has quit [Ping timeout: 252 seconds] 21:17 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has joined #bitcoin-core-dev 21:29 -!- murrayn [~dafuq@unaffiliated/murrayn] has quit [Quit: Adios mofos] 21:29 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has quit [Remote host closed the connection] 21:30 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has joined #bitcoin-core-dev 21:31 -!- Krellan [~Krellan@71.136.136.149] has quit [Remote host closed the connection] 21:37 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has joined #bitcoin-core-dev 21:42 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 21:45 -!- Jmabsd [~jmabsd@1.186.68.250] has quit [Ping timeout: 272 seconds] 21:46 -!- murrayn [~dafuq@unaffiliated/murrayn] has joined #bitcoin-core-dev 21:48 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 21:49 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 21:53 -!- phwalkr [~phwalkr@2001:1284:f01c:5d1e:d089:d43e:883d:be7a] has quit [Ping timeout: 264 seconds] 21:54 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 22:14 < luke-jr> was there a point to https://github.com/bitcoin-core/leveldb/pull/19 considering we limit leveldb to 1000 open files anyway? 22:17 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 22:19 < sipa> luke-jr: mmap'ed files aren't kept open after creating the mapping 22:19 < sipa> so they don't count towards the 1000 files limit iirc 22:24 < luke-jr> sipa: but we increased the file limit to 1000 *because* mmaps didn't count toward the fd limit 22:24 < luke-jr> the limit I'm talking about is max_open_files, not ulimit 22:32 -!- Taifta [~Taifta@141.226.151.137] has quit [Ping timeout: 268 seconds] 22:35 < sipa> luke-jr: yes, but there is a separate mmap limit it seems 22:51 -!- TheHoliestRoger [~TheHolies@unaffiliated/theholiestroger] has quit [Quit: Find me in #TheHolyRoger or https://theholyroger.com] 22:53 -!- TheHoliestRoger [~TheHolies@unaffiliated/theholiestroger] has joined #bitcoin-core-dev 22:59 -!- TheHoliestRoger [~TheHolies@unaffiliated/theholiestroger] has quit [Quit: Find me in #TheHolyRoger or https://theholyroger.com] 23:02 -!- TheHoliestRoger [~TheHolies@unaffiliated/theholiestroger] has joined #bitcoin-core-dev 23:02 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 268 seconds] 23:02 -!- ken2812221 [~ken281222@180.217.87.172] has joined #bitcoin-core-dev 23:03 -!- Krellan [~Krellan@2601:640:4000:9258:9100:ec7b:d9b9:b50c] has joined #bitcoin-core-dev 23:04 -!- Krellan_ [~Krellan@2601:640:4000:9258:6159:ba39:f506:92bc] has joined #bitcoin-core-dev 23:05 -!- nOgAnOo_ [sid146237@gateway/web/irccloud.com/x-cobpzeohazarakez] has quit [Ping timeout: 260 seconds] 23:06 -!- d_t [~d_t@108-65-77-11.lightspeed.sntcca.sbcglobal.net] has joined #bitcoin-core-dev 23:07 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has quit [Ping timeout: 252 seconds] 23:08 -!- Krellan [~Krellan@2601:640:4000:9258:9100:ec7b:d9b9:b50c] has quit [Ping timeout: 264 seconds] 23:13 < luke-jr> sipa: max_open_files effectively limits the number of mmaps too 23:13 < luke-jr> so it seems like the LevelDB mmap limit change is effectively a no-op since we keep the 1000 max_open_files limit :x 23:14 < gmaxwell> luke-jr: it's very much not a no op change. 23:15 < luke-jr> gmaxwell: how does it have an effect? 23:15 < gmaxwell> it causes more mmaps to be open, and less files. you can see this easily with lsof. 23:16 < luke-jr> the mmap limit is enforced by NewRandomAccessFile, which is only ever called by TableCache which enforces the max_open_files limit :/ 23:16 < gmaxwell> Well I'm not telling you what the source code says, I'm telling you what it does. :P 23:16 < luke-jr> is there a bug in the LRU cache stuff? :/ 23:18 < luke-jr> gmaxwell: is there an easy way to reproduce what you observe? 23:19 < gmaxwell> yea, start a node up no connect, with and without the change and lsof it and look at the number of mmaped files. you might need to crank checkblocks up a bit. 23:21 -!- Giszmo [~leo@168.196.201.82] has joined #bitcoin-core-dev 23:32 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has quit [Ping timeout: 252 seconds] 23:33 -!- Giszmo [~leo@168.196.201.82] has quit [Ping timeout: 272 seconds] 23:35 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 23:36 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 23:46 -!- plankers [~plank@38.87.81.82] has joined #bitcoin-core-dev 23:53 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has joined #bitcoin-core-dev 23:57 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 264 seconds] 23:57 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Read error: Connection reset by peer] 23:58 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev --- Log closed Sun Aug 26 00:00:50 2018