--- Log opened Sat Aug 25 00:00:33 2018
--- Day changed Sat Aug 25 2018
00:00 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has quit [Remote host closed the connection]
00:00 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has joined #bitcoin-core-dev
00:18 -!- Lauda [~quassel@unaffiliated/lauda] has joined #bitcoin-core-dev
00:29 -!- vexbuy [~vexbuy@89.39.107.192] has joined #bitcoin-core-dev
00:37 -!- murrayn [~dafuq@unaffiliated/murrayn] has quit [Read error: Connection reset by peer]
00:39 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev
00:39 -!- wxss [~user@5.187.21.78] has joined #bitcoin-core-dev
00:43 -!- promag [~promag@83.223.235.81] has quit [Ping timeout: 244 seconds]
00:53 -!- Rootsudo [~textual@180.190.116.243] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
00:54 -!- Rootsudo [~textual@180.190.116.243] has joined #bitcoin-core-dev
00:55 -!- Rootsudo [~textual@180.190.116.243] has quit [Client Quit]
00:55 -!- Rootsudo [~textual@180.190.116.243] has joined #bitcoin-core-dev
00:55 -!- Rootsudo [~textual@180.190.116.243] has quit [Client Quit]
00:57 -!- Rootsudo [~textual@180.190.116.243] has joined #bitcoin-core-dev
00:57 -!- Rootsudo [~textual@180.190.116.243] has quit [Client Quit]
01:04 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has joined #bitcoin-core-dev
01:05 -!- vexbuy [~vexbuy@89.39.107.192] has quit [Remote host closed the connection]
01:07 -!- vexbuy [~vexbuy@89.39.107.192] has joined #bitcoin-core-dev
01:08 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has joined #bitcoin-core-dev
01:11 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has quit [Quit: Leaving.]
01:12 -!- vexbuy [~vexbuy@89.39.107.192] has quit [Ping timeout: 268 seconds]
01:34 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection]
01:35 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev
01:40 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has quit [Remote host closed the connection]
01:42 -!- vexbuy [~vexbuy@89.39.107.192] has joined #bitcoin-core-dev
01:42 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has joined #bitcoin-core-dev
02:08 -!- csknk [~csknk@unaffiliated/csknk] has joined #bitcoin-core-dev
02:22 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev
02:22 -!- promag [~promag@83.223.235.81] has quit [Remote host closed the connection]
02:37 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has quit [Ping timeout: 252 seconds]
03:03 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
03:03 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection]
03:06 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
03:22 -!- profmac [~ProfMac@2001:470:1f0f:226:21c7:9b4e:db96:accb] has quit [Ping timeout: 264 seconds]
03:30 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev
03:35 -!- promag [~promag@83.223.235.81] has quit [Ping timeout: 268 seconds]
03:45 -!- cdecker [~cdecker@mail.snyke.net] has quit [Quit: ZNC - http://znc.in]
04:27 -!- vexbuy [~vexbuy@89.39.107.192] has quit [Remote host closed the connection]
04:34 -!- goatpig [56eece80@gateway/web/freenode/ip.86.238.206.128] has quit [Ping timeout: 252 seconds]
04:40 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
04:42 -!- promag [~promag@83.223.234.98] has joined #bitcoin-core-dev
04:42 -!- IGHOR [~quassel@93.178.216.72] has quit [Read error: No route to host]
04:43 -!- IGHOR [~quassel@93.178.216.72] has joined #bitcoin-core-dev
04:46 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit []
04:46 -!- promag [~promag@83.223.234.98] has quit [Ping timeout: 252 seconds]
05:04 -!- promag [~promag@83.223.234.98] has joined #bitcoin-core-dev
05:11 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
05:36 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection]
05:37 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev
05:41 -!- as1nc [~as1nc@ax213-1-82-66-157-194.fbx.proxad.net] has joined #bitcoin-core-dev
05:41 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection]
05:42 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
05:46 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Ping timeout: 252 seconds]
05:47 -!- promag [~promag@83.223.234.98] has quit [Remote host closed the connection]
05:48 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
05:56 -!- esotericnonsense [~esotericn@unaffiliated/esotericnonsense] has quit [Ping timeout: 268 seconds]
06:25 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
06:39 -!- promag [~promag@83.223.234.98] has joined #bitcoin-core-dev
06:40 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 272 seconds]
06:43 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Read error: Connection reset by peer]
06:44 -!- promag [~promag@83.223.234.98] has quit [Ping timeout: 268 seconds]
06:44 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev
06:55 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
06:56 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection]
06:58 -!- owowo [~ovovo@unaffiliated/ovovo] has quit [Ping timeout: 264 seconds]
07:06 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
07:08 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 272 seconds]
07:11 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 244 seconds]
07:13 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
07:23 -!- owowo [~ovovo@unaffiliated/ovovo] has joined #bitcoin-core-dev
07:30 -!- promag [~promag@83.223.234.98] has joined #bitcoin-core-dev
07:43 -!- promag [~promag@83.223.234.98] has quit [Remote host closed the connection]
08:03 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection]
08:04 -!- vexbuy [~vexbuy@89.39.107.194] has joined #bitcoin-core-dev
08:04 -!- bsm117532 [~mcelrath@c-24-61-184-150.hsd1.ma.comcast.net] has joined #bitcoin-core-dev
08:07 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
08:08 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 268 seconds]
08:12 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 252 seconds]
08:17 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
08:20 -!- vexbuy [~vexbuy@89.39.107.194] has quit [Ping timeout: 244 seconds]
08:22 -!- ossifrage [~ossifrage@unaffiliated/ossifrage] has quit [Remote host closed the connection]
08:23 -!- ossifrage [~ossifrage@unaffiliated/ossifrage] has joined #bitcoin-core-dev
08:40 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
09:16 -!- drexl [~drexl@cpc130676-camd16-2-0-cust445.know.cable.virginm.net] has joined #bitcoin-core-dev
09:33 -!- Zenton [~user@54.104.135.37.dynamic.jazztel.es] has quit [Ping timeout: 244 seconds]
09:38 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection]
09:39 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev
09:41 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has joined #bitcoin-core-dev
09:49 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has quit [Ping timeout: 276 seconds]
09:55 -!- CubicEarth [~CubicEart@c-73-181-185-197.hsd1.wa.comcast.net] has quit [Ping timeout: 260 seconds]
10:02 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
10:13 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has joined #bitcoin-core-dev
10:15 -!- spinza [~spin@155.93.246.187] has quit [Ping timeout: 244 seconds]
10:26 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev
10:35 -!- Taifta [~Taifta@141.226.151.137] has joined #bitcoin-core-dev
10:45 -!- jhfrontz [~Adium@cpe-184-57-118-36.columbus.res.rr.com] has quit [Quit: Leaving.]
10:48 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
10:49 -!- jhfrontz [~Adium@cpe-184-57-118-36.columbus.res.rr.com] has joined #bitcoin-core-dev
11:08 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds]
11:20 -!- drexl [~drexl@cpc130676-camd16-2-0-cust445.know.cable.virginm.net] has quit [Read error: Connection reset by peer]
11:29 -!- csknk [~csknk@unaffiliated/csknk] has quit [Quit: leaving]
11:51 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has quit [Remote host closed the connection]
11:58 -!- davex__ [~user@67-3-156-44.omah.qwest.net] has joined #bitcoin-core-dev
12:03 -!- hebasto [~hebasto@195.60.70.234] has joined #bitcoin-core-dev
12:07 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has joined #bitcoin-core-dev
12:10 -!- ken2812221 [~ken281222@110.50.144.85] has quit [Ping timeout: 252 seconds]
12:21 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection]
12:22 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev
12:32 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has quit [Remote host closed the connection]
12:32 -!- wumpus [~wumpus@pdpc/supporter/professional/wumpus] has joined #bitcoin-core-dev
13:39 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Ping timeout: 244 seconds]
13:39 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
13:39 < jamesob> re: memory usage increase: preliminary bisections are in and MarcoFalke and I are betting it's the leveldb changes. https://i.imgur.com/8aXRzwe.png
13:41 -!- intcat [~zshlyk@gateway/tor-sasl/intcat] has quit [Ping timeout: 250 seconds]
13:42 < gmaxwell> jamesob: wait. how are we measuring memory usage in that benchmark?
13:42 < jamesob> gmaxwell: time -f %M (ie resident set size)
13:49 < jamesob> https://github.com/chaincodelabs/bitcoinperf/blob/master/runner/main.py#L659-L668
13:53 < sipa> so every mmap causes 150 kB RSS increase? :s
13:54 < gmaxwell> no because we're not actually using the new maximum.
13:54 < gmaxwell> (I mean not using all of)
13:54 < sipa> aha.
13:54 < gmaxwell> So even MOAR.
13:55 < gmaxwell> which is suspect
13:55 < sipa> mmap by default will prefetch mapped pages into memory
13:55 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has quit [Ping timeout: 272 seconds]
13:55 < sipa> by default, 31 4kB blocks
13:55 < gmaxwell> oh interesting. but those pages are clean they'll just get evicted, they really shouldn't be counted bt RSS :(
13:57 < gmaxwell> so MADV_RANDOM
13:57 < gmaxwell> ...
13:57 < sipa> madvice(MADV_DONTNEED) will disable the reading entirely
13:58 < sipa> which may be useful to diagnose the issue
13:58 < jamesob> (for those following along at home: https://github.com/bitcoin/bitcoin/pull/13925/files)
13:58 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection]
13:59 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
13:59 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
13:59 < gmaxwell> Also setting the maximum maps really low, like..2 might be interesting.
14:02 < gmaxwell> but if this is the problem, MADV_RANDOM is probably the fix to the extent that its an actual problem at all.
14:02 < gmaxwell> Though we should do a reindex benchmark to make sure MADV_RANDOM doesn't hurt performance.
14:03 < wumpus> PSA: if after the latest merge you get a linker error "/usr/local/include/boost/smart_ptr/shared_ptr.hpp:728: undefined reference to `translationInterface", you need to do a 'make clean' and re-do the make and it will work
14:03 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 268 seconds]
14:04 < luke-jr> wumpus: is there a reason the gitian linux yml has g++-riscv64-linux-gnu as a dep? seems to pull in GCC 7 when we're using GCC 8 now?
14:04 < luke-jr> wumpus: if `make clean` ever fixes something, that means there's a bug in the build system :/
14:06 < sipa> gmaxwell, jamesob: LMDB uses MADV_RANDOM it seems
14:06 < sipa> (though its design is different, i don't know their access patterns)
14:06 < wumpus> luke-jr: yes, it must be missing some changes in dependency detection between source and header files (another one is if you change something in univalue, it won't detect it)
14:06 < sipa> jamesob: try this:
14:06 < sipa> void* base = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
14:06 < sipa> if (base != MAP_FAILED) {
14:06 < sipa> + madvise(base, size, MADV_DONTNEED);
14:07 < sipa> in src/leveldb/util/env_posix.cc
14:07 < sipa> (and/or MADV_RANDOM)
14:07 < wumpus> luke-jr: it pulls in both gcc 7 and 8, I think that's necessary due to some strangeness with the packages (some symlink will only exist when g++-riscv64-linux-gnu is also installed)
14:07 < luke-jr> DONTNEED sounds wrong?
14:07 < luke-jr> wumpus: ah, weird
14:08 < wumpus> luke-jr: you might be able to get around it, but I noticed and tried as well and ran into a dead end
14:08 < sipa> luke-jr: to diagnoze
14:08 < sipa> luke-jr: it would be interesting to see what the effect on RSS is with DONTNEED, to have an idea to what extent our memory usage is due to mmap caching
14:09 < jamesob> sipa: giving it a shot now
14:09 < wumpus> luke-jr: at least it's not MADV_HWPOISON!
14:10 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev
14:10 -!- intcat [~zshlyk@gateway/tor-sasl/intcat] has joined #bitcoin-core-dev
14:10 -!- plankers [~plank@200.7.98.14] has joined #bitcoin-core-dev
14:10 < luke-jr> wumpus: lol
14:11 < gmaxwell> It's plausable to me that MADV_RANDOM will help performance or at least be neutral.
14:11 < wumpus> yes, to me too
14:11 < wumpus> our access pattern is more or less random
14:12 < gmaxwell> I don't recall now, though I know I researched this before... does leveldb's bisection interpolate assuming keys are uniform and that their values are uniformly sized or does it plain bisect?
14:12 < wumpus> (except in the rare times it's iterating over the whole utxo set in order, like when computing statistics)
14:13 < gmaxwell> esp in the case of plain bisection, prefetching is a bad behavior.
14:13 < wumpus> I don't know
14:13 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection]
14:14 -!- vexbuy [~vexbuy@217.23.3.171] has joined #bitcoin-core-dev
14:15 < sipa> gmaxwell: there is an index at the end of each ldb file
14:15 < sipa> i assume it bisects in that index in a naive way, but i'm not sure
14:21 < jamesob> sipa: bench is running; we'll know how your change works in about six hours
14:22 < sipa> jamesob: awesome
14:22 < luke-jr> hm, I think I will regret relatime when I try to prune gitian caches
14:53 -!- meshcollider_ [uid246294@gateway/web/irccloud.com/x-baedxxadrwiqngep] has joined #bitcoin-core-dev
14:54 -!- Amuza [~amuza@212.230.156.111] has joined #bitcoin-core-dev
14:55 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
14:56 -!- vexbuy [~vexbuy@217.23.3.171] has quit [Read error: Connection reset by peer]
15:02 -!- jhfrontz [~Adium@cpe-184-57-118-36.columbus.res.rr.com] has quit [Read error: Connection reset by peer]
15:08 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
15:09 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer]
15:11 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev
15:17 -!- plankers [~plank@200.7.98.14] has quit [Ping timeout: 264 seconds]
15:29 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer]
15:32 -!- plankers [~plank@38.87.81.82] has joined #bitcoin-core-dev
15:32 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev
15:33 -!- Amuza [~amuza@212.230.156.111] has quit [Ping timeout: 252 seconds]
15:48 -!- esotericnonsense [~esotericn@unaffiliated/esotericnonsense] has joined #bitcoin-core-dev
15:57 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...]
16:07 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has quit [Ping timeout: 250 seconds]
16:11 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has joined #bitcoin-core-dev
16:18 -!- dcousens [~dcousens@110.140.174.10] has quit [Ping timeout: 255 seconds]
16:23 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection]
16:24 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev
16:24 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev
16:31 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 268 seconds]
16:40 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has quit [Ping timeout: 276 seconds]
16:41 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has joined #bitcoin-core-dev
17:03 -!- meshcollider_ [uid246294@gateway/web/irccloud.com/x-baedxxadrwiqngep] has quit [Quit: Connection closed for inactivity]
17:09 -!- plankers [~plank@38.87.81.82] has quit [Ping timeout: 252 seconds]
17:15 < gmaxwell> sipa: ah that might have been why I didn't bother trying to make it interpolate-- the seeks are in an index.
17:22 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has quit [Ping timeout: 252 seconds]
17:25 -!- plankers [~plank@200.7.98.14] has joined #bitcoin-core-dev
17:29 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has joined #bitcoin-core-dev
17:30 -!- Krellan [~Krellan@2601:640:4000:9258:dc59:c5c3:66e6:7965] has quit [Remote host closed the connection]
17:46 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
17:53 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds]
18:31 -!- plankers [~plank@200.7.98.14] has quit [Ping timeout: 272 seconds]
19:34 -!- Jmabsd [~jmabsd@1.186.68.250] has joined #bitcoin-core-dev
19:38 < jamesob> sipa gmaxwell: mem usage increase is for sure the leveldb change (https://i.imgur.com/yMHJI37.png) and sipa's change didn't change the elevated usage
19:39 < sipa> jamesob: so it's still elevated with my patch?
19:39 < sipa> or it's not elevated anymore?
19:40 < jamesob> still elevated :(
19:41 < gmaxwell> might want to make a microbenchmar to figure out if its mmap behavior or something else in leveldb.
19:42 < gmaxwell> also because it'll be easier to expirement (e.g. see if MAP_NORESERVE does anything) when we don't have to wait 5 hours.
19:42 < gmaxwell> just a loop to mmap all the files in a directory then sleep 30 seconds should test the earlier hypothesis.
19:42 -!- Krellan [~Krellan@71.136.136.149] has joined #bitcoin-core-dev
19:43 < jamesob> gmaxwell: yup, agree
19:43 < jamesob> will write if I have time tomorrow
19:50 -!- murrayn [~dafuq@unaffiliated/murrayn] has joined #bitcoin-core-dev
20:12 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer]
20:13 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev
20:16 -!- phwalkr [~phwalkr@2001:1284:f01c:5d1e:d089:d43e:883d:be7a] has joined #bitcoin-core-dev
20:20 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
20:22 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection]
20:28 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev
20:28 -!- Rootsudo [~textual@180.190.116.181] has joined #bitcoin-core-dev
20:30 -!- plankers [~plank@200.7.98.14] has joined #bitcoin-core-dev
20:38 -!- plankers [~plank@200.7.98.14] has quit [Read error: No route to host]
20:46 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection]
20:47 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
20:51 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Ping timeout: 252 seconds]
20:53 < ossifrage> It seems like MADV_RANDOM is useful if there is less locality to the reads then the size of the prefetch window, but I was under the impression that the prefetch is triggered by the pagefault and not by the mmap() so you actually have to do reads to get the extra pages
21:06 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has quit [Ping timeout: 252 seconds]
21:17 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has joined #bitcoin-core-dev
21:29 -!- murrayn [~dafuq@unaffiliated/murrayn] has quit [Quit: Adios mofos]
21:29 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has quit [Remote host closed the connection]
21:30 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has joined #bitcoin-core-dev
21:31 -!- Krellan [~Krellan@71.136.136.149] has quit [Remote host closed the connection]
21:37 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
21:42 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
21:45 -!- Jmabsd [~jmabsd@1.186.68.250] has quit [Ping timeout: 272 seconds]
21:46 -!- murrayn [~dafuq@unaffiliated/murrayn] has joined #bitcoin-core-dev
21:48 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer]
21:49 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev
21:53 -!- phwalkr [~phwalkr@2001:1284:f01c:5d1e:d089:d43e:883d:be7a] has quit [Ping timeout: 264 seconds]
21:54 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection]
22:14 < luke-jr> was there a point to https://github.com/bitcoin-core/leveldb/pull/19 considering we limit leveldb to 1000 open files anyway?
22:17 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
22:19 < sipa> luke-jr: mmap'ed files aren't kept open after creating the mapping
22:19 < sipa> so they don't count towards the 1000 files limit iirc
22:24 < luke-jr> sipa: but we increased the file limit to 1000 *because* mmaps didn't count toward the fd limit
22:24 < luke-jr> the limit I'm talking about is max_open_files, not ulimit
22:32 -!- Taifta [~Taifta@141.226.151.137] has quit [Ping timeout: 268 seconds]
22:35 < sipa> luke-jr: yes, but there is a separate mmap limit it seems
22:51 -!- TheHoliestRoger [~TheHolies@unaffiliated/theholiestroger] has quit [Quit: Find me in #TheHolyRoger or https://theholyroger.com]
22:53 -!- TheHoliestRoger [~TheHolies@unaffiliated/theholiestroger] has joined #bitcoin-core-dev
22:59 -!- TheHoliestRoger [~TheHolies@unaffiliated/theholiestroger] has quit [Quit: Find me in #TheHolyRoger or https://theholyroger.com]
23:02 -!- TheHoliestRoger [~TheHolies@unaffiliated/theholiestroger] has joined #bitcoin-core-dev
23:02 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 268 seconds]
23:02 -!- ken2812221 [~ken281222@180.217.87.172] has joined #bitcoin-core-dev
23:03 -!- Krellan [~Krellan@2601:640:4000:9258:9100:ec7b:d9b9:b50c] has joined #bitcoin-core-dev
23:04 -!- Krellan_ [~Krellan@2601:640:4000:9258:6159:ba39:f506:92bc] has joined #bitcoin-core-dev
23:05 -!- nOgAnOo_ [sid146237@gateway/web/irccloud.com/x-cobpzeohazarakez] has quit [Ping timeout: 260 seconds]
23:06 -!- d_t [~d_t@108-65-77-11.lightspeed.sntcca.sbcglobal.net] has joined #bitcoin-core-dev
23:07 -!- Giszmo [~leo@pc-72-54-46-190.cm.vtr.net] has quit [Ping timeout: 252 seconds]
23:08 -!- Krellan [~Krellan@2601:640:4000:9258:9100:ec7b:d9b9:b50c] has quit [Ping timeout: 264 seconds]
23:13 < luke-jr> sipa: max_open_files effectively limits the number of mmaps too
23:13 < luke-jr> so it seems like the LevelDB mmap limit change is effectively a no-op since we keep the 1000 max_open_files limit :x
23:14 < gmaxwell> luke-jr: it's very much not a no op change.
23:15 < luke-jr> gmaxwell: how does it have an effect?
23:15 < gmaxwell> it causes more mmaps to be open, and less files. you can see this easily with lsof.
23:16 < luke-jr> the mmap limit is enforced by NewRandomAccessFile, which is only ever called by TableCache which enforces the max_open_files limit :/
23:16 < gmaxwell> Well I'm not telling you what the source code says, I'm telling you what it does. :P
23:16 < luke-jr> is there a bug in the LRU cache stuff? :/
23:18 < luke-jr> gmaxwell: is there an easy way to reproduce what you observe?
23:19 < gmaxwell> yea, start a node up no connect, with and without the change and lsof it and look at the number of mmaped files. you might need to crank checkblocks up a bit.
23:21 -!- Giszmo [~leo@168.196.201.82] has joined #bitcoin-core-dev
23:32 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has quit [Ping timeout: 252 seconds]
23:33 -!- Giszmo [~leo@168.196.201.82] has quit [Ping timeout: 272 seconds]
23:35 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer]
23:36 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev
23:46 -!- plankers [~plank@38.87.81.82] has joined #bitcoin-core-dev
23:53 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has joined #bitcoin-core-dev
23:57 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 264 seconds]
23:57 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Read error: Connection reset by peer]
23:58 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev
--- Log closed Sun Aug 26 00:00:50 2018