--- Day changed Sat Aug 27 2016
00:05  * luke-jr ponders if iotop would report writing if it was swapping other processes to disk to do caching for us
00:09 < wumpus> yes it reports swapping as writing, but wouldn't account swapping of *other* processes to disk to bitcoind
00:09 < wumpus> IIRC there's a kswapd that gets all the blame for swapping
00:09 < luke-jr> hmm
00:10 < luke-jr> it did pick up speed and finished in 15 mins (just now) fwiw
00:10 < luke-jr> didn't see any writing this time either
00:11 < wumpus> phew
00:12 < luke-jr> but I guess I did manage to reproduce https://www.reddit.com/r/Bitcoin/comments/4zrxs1/qtcore_client_taking_ages_to_start/ after all
00:12 < luke-jr> just a matter of dropping caches
00:12 -!- OxADADA_ [~OxADADA@alumni-linux.ccs.neu.edu] has joined #bitcoin-core-dev
00:12 < luke-jr> gmaxwell: can you confirm on your slow laptop? echo 3 > /proc/sys/vm/drop_caches
00:14 < wumpus> usually if the client takes ages to start there's a backlog of blocks to verify
00:14 -!- asoltys_ [~bitcoinco@23.94.96.232] has joined #bitcoin-core-dev
00:14 -!- Pasha [~C@unaffiliated/cory] has joined #bitcoin-core-dev
00:15 < sipa> question: how many times has the initial verification at startup actually caught corruption
00:15 < sipa> nobody knows, of course
00:15 < wumpus> I don't know - but I think it would be just as effective to just check a few blocks
00:15 < luke-jr> sipa: not sure if it's still a problem, but every time when we had those powerfail-corrupts-db problem? (unless that was caught by something else?)
00:16 < wumpus> the latest blocks are the most likely to be corrupt and below that it drops off 
00:16 < sipa> luke-jr: those result in a leveldb checksum error
00:16  * luke-jr didn't realise the slowish startup time because he had checkblocks=4 checklevel=6 in his bitcoin.conf
00:16 < sipa> there are only 4 checklevels :)
00:16 < wumpus> luke-jr: same here - I think the default checkblocks should be much lower
00:16 < luke-jr> well, if there's ever more added, I'm ready! :P
00:16 -!- owowo [ovovo@gateway/vpn/mullvad/x-hlopwijsonuzqriv] has quit [Ping timeout: 240 seconds]
00:16 -!- ratoder [~ratoder@static.111.19.201.138.clients.your-server.de] has quit [Ping timeout: 240 seconds]
00:16 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has quit [Ping timeout: 240 seconds]
00:16 -!- OxADADA [~OxADADA@alumni-linux.ccs.neu.edu] has quit [Ping timeout: 240 seconds]
00:16 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has quit [Ping timeout: 240 seconds]
00:16 -!- nobits [~nobits@rrmmtl.ca] has quit [Ping timeout: 240 seconds]
00:16 -!- gluytium [~g@45.63.97.181] has quit [Ping timeout: 240 seconds]
00:16 -!- Cory [~C@unaffiliated/cory] has quit [Ping timeout: 240 seconds]
00:16 -!- asoltys [~bitcoinco@23.94.96.232] has quit [Remote host closed the connection]
00:16 < wumpus> a as-thorough-as-possible check on just a few blocks
00:16 < sipa> and 4 blocks is not very much
00:17 < wumpus> well, take 10 then
00:17 -!- nobits [~nobits@rrmmtl.ca] has joined #bitcoin-core-dev
00:17 -!- owowo [ovovo@gateway/vpn/mullvad/x-gszoztcpukikdfaq] has joined #bitcoin-core-dev
00:17 < luke-jr> sipa: my PC is on UPS ;)
00:17 < sipa> wumpus: jonasschnelli has a patch to switch to txcount based limiting
00:18 < luke-jr> hmm, that's an interesting idea.
00:18 < wumpus> that's good, but I think the effective default check depth should also be lowered, don't know if it does that
00:18 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has joined #bitcoin-core-dev
00:18 < wumpus> or maybe write a flag on 'clean' shutdown, and do a reduced check in that case?
00:18 < luke-jr> set it for the equivalent checkblocks back when it was introduced? :p
00:18 -!- gluytium [~g@45.63.97.181] has joined #bitcoin-core-dev
00:19 < sipa> wumpus: it sets the default to 100000 txn
00:19 < sipa> i guess it can be lower even
00:19 < wumpus> sipa: ok
00:19 < sipa> but it also insists on at least 6 blocks
00:19 < gmaxwell> we need to do something about the startup check. 
00:20 < gmaxwell> unfortunately just checking a couple blocks is not much of a test, but anything more takes too long for most people.
00:20 < sipa> wumpus: interesting idea
00:20 < luke-jr> I wonder how difficult it would be to background it
00:20 < gmaxwell> wumpus: neat idea!
00:21 -!- Pasha is now known as Cory
00:22 < gmaxwell> when we were having windows corruption, what was needed to reliably detect that?
00:22 < wumpus> just opening the leveldb IIRC
00:22 < btcdrak> roasbeef: thanks!!
00:23 < wumpus> or maybe the first access. No thorough checking was necessary
00:23  * sipa suggests: 10000 txn (which corresponds to ~6 blocks currently)
00:24 < wumpus> sounds good to me
00:25 < wumpus> it's very helpful that leveldb has its own corruption detection here
00:25 < gmaxwell> why bother with the txn count?
00:25 < gmaxwell> just set it to 6 blocks?
00:25 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev
00:25 < wumpus> because that auto-adapts
00:25 < gmaxwell> so? other than segwit the txn count of blocks won't change much.
00:25 < wumpus> if blocks grow again, it won't become  slower
00:26 < wumpus> heh yes that's a completely different discission
00:26 < sipa> gmaxwell: if you are early in IBD you probably want more blockss
00:26 < gmaxwell> sipa: okay, I'll but that.. but kind of a corner case.
00:26 < wumpus> but the fact is that transaction could is a better measure of run time
00:26 < wumpus> count*
00:26 < gmaxwell> I was at the verge of suggesting this just do a single block.
00:26 < wumpus> and amount of data checked
00:27 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-cpnrwcanwjjotelf] has joined #bitcoin-core-dev
00:27 < sipa> heh, fine by me as well
00:27 < gmaxwell> It's not yielding a high payoff in detected errors, the ones I know it detected (chainstate version corruption, need checking back to your last restat to reliably report)
00:27 < wumpus> well the first priority is to make the insane wait time go away
00:28 < wumpus> whether the new number becomes low or ultra-low is less important :)
00:28 < gmaxwell> One or two block keeps the code live and working, I think this code is useful around refactorings and changes to the code.
00:48 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
00:48 < btcdrak> i guess the reply to Roger's testnet response is no, BU runs in production and is a menace to the network
00:48 < btcdrak> BU has no peer review and no chance in hell of being used for real
00:49 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
01:06 < btcdrak> wumpus: rebased #7562
01:07 < sipa> btcdrak: what was the problem with the tests?
01:07 < btcdrak> gmaxwell: can you explain a bit more about your suggestion regarding my attempt at extracting the MacOSX sdk on linux?
01:09 < btcdrak> sipa: changing the defaul tx version affected a couple of tests that were comparing hash values, or sorting txs by hash. obviously the hashes change when tx version is bumped. Those seem innocuous. There is one test however that I dont understand which I commented on. Not sure why it is affected by changing the version number.
01:10 < btcdrak> I dont know if it's the tests' fault and innocuous, or revealing an issue.
01:15 -!- Megaf [~quassel@unaffiliated/megaf] has joined #bitcoin-core-dev
01:20 -!- fengling [~fengling@58.135.95.136] has quit [Ping timeout: 240 seconds]
01:22 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)]
01:30 < gmaxwell> btcdrak: so, first someone extracts it via OSX.  Then we take the extracted binary and find all the offsets for its data in the decompressed file. 
01:31 < gmaxwell> we can distribute the offset list.
01:31 < gmaxwell> actually, a took like xdelta or another binary diff tool might just handle it.
01:34 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has joined #bitcoin-core-dev
01:40 < sipa> but the dmg file is compressed
01:40 < sipa> can we decompress it?
01:41 < sipa> without "installing", that is
01:41 < gmaxwell> it's a 7z file according to drak
01:43 < gmaxwell> so I was assuming that.
01:43 < btcdrak> gmaxwell: one thing I am not sure about is if 7z is actually extracting it correctly. It seems to be, and the bug seems more like in the linux implementation of hfsplus; but it is feasible that the compression algo was tweaked and 7z is unaware 
01:43 < gmaxwell> unlikely, 7z has checksums.
01:47 < sipa> so we'd xdelta the 7z-decompressed dmg with the resulting compressed .tar?
01:47 < gmaxwell> uncompressed tar.
01:47 < gmaxwell> oh what we want is a tgz.
01:47 < gmaxwell> yes.
01:47 < gmaxwell> so the file we normally get out of it.
01:48 < gmaxwell> and if the resulting delta is small, we call it done.
01:48 < luke-jr> btcdrak: what version of 7zip did you use?
01:48 < sipa> btcdrak: where is hfsplus used?
01:49 < btcdrak> luke-jr: version 9.20 
01:49 < luke-jr> hmm, it tells me: Error: Can not open file as archive
01:51 -!- fengling [~fengling@58.135.95.136] has joined #bitcoin-core-dev
01:56 < btcdrak> sipa: https://www.irccloud.com/pastebin/S3zIzO3D/
01:58 < luke-jr> oh, my DMG file is truncated
01:59 < btcdrak> so basically the files we want are in Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk, but using the method above you get a bunch of empty files. On a Mac you run "tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk"
01:59 < btcdrak> cfields_ documented it here https://github.com/bitcoin/bitcoin/blob/master/doc/README_osx.md
02:00 < sipa> but we don't need all of the contents of that dmg, right?
02:00 < luke-jr> not nearly
02:00 < luke-jr> but it's copyrighted and non-redistributable :<
02:00 < btcdrak> sipa right, we just need the files from Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/* which we turn into a tar.gz
02:00 < sipa> i see
02:00 < btcdrak> That gives us 47MB tar.gz :-p
02:02 < btcdrak> we drop that in the gitian-builder/inputs/ folder when performing the Gitian rituals. 
02:02 < sipa> got it
02:02  * luke-jr waits on copy over LAN.. Xcode_7.3.1.dmg   27% 1383MB   6.5MB/s   09:08 ETA
02:03 < sipa> 14m here
02:03 < luke-jr> you and your fast internet :x :p
02:03 < sipa> luke-jr: this is over t-mobile roaming data
02:04 -!- fengling [~fengling@58.135.95.136] has quit [Ping timeout: 240 seconds]
02:04 < luke-jr> sadly, T-Mobile data here seems to only be about 5 Mbps
02:10 < gmaxwell> 2hr 49m remaining here.
02:10 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
02:14 < btcdrak> RIP sipa's roaming data charges
02:14 < sipa> btcdrak: it's free
02:14 < sipa> it's effectively cheaper than my swiss wired or wireless internet
02:15 < sipa> s/free/fixed price/
02:16 < gmaxwell> btcdrak: a while back tmobile ceo put out some open letter about "data abusers" and for some reason it said, 'mining bitcoin' -- for a while we wondered if perhaps he was talking about blockstream. https://newsroom.t-mobile.com/news-and-blogs/stopping-network-abusers.htm
02:16 < gmaxwell> apparently blockstream was using several TB a month on tmobile.
02:18 < gmaxwell> doesn't everyone run a full node on their phone?
02:18 < luke-jr> lol, xdelta is 45 MB
02:18 < sipa> :(
02:19 < sipa> luke-jr: how did you decompress the dmg?
02:19 < luke-jr> 7z
02:19 < gmaxwell> lol
02:19 < gmaxwell> might need more options for it to actually find the match
02:20 < luke-jr> maybe there's some tweaking to the tar we need to put it in a better order?
02:20 -!- fengling [~fengling@58.135.95.136] has joined #bitcoin-core-dev
02:20 < gmaxwell> yes, that too.
02:20 < luke-jr> oh wait
02:20 < luke-jr> hmm, nm
02:20 < luke-jr> the uncompressed tar is 306 MB FWIW
02:20 < luke-jr> but xdelta's patch is compressed.
02:20 < gmaxwell> were you using the compressed tar as the input? 
02:20 < luke-jr> no
02:21 < sipa> hpmount, mount -t hfsplus, 7zr... all fail to open the file
02:21 < gmaxwell> maybe we should just encrypt the damn tar with the whole dmg file as the key, and then call that sufficient for legal purposes. :P
02:22 < luke-jr> sipa: need 7z, not 7zr
02:22 < luke-jr> 7zr has formats removed
02:22 < luke-jr> gmaxwell: hmm, I wonder if that'd be okay
02:22 < luke-jr> I turned off xdelta's patch compression and it's 261 MB :/
02:23 < gmaxwell> smaller than 300 so it found something.
02:23 < gmaxwell> luke-jr: I think doing that would be defensible.
02:23 < luke-jr> xdelta3: warning: input position 310378496 overflowed instruction buffer, needed 43091 (vs. 32768), consider changing -I
02:24 < gmaxwell> consider changing -I
02:24  * luke-jr sets -I to unlimited
02:24 < luke-jr> not doing much better :<
02:24 < luke-jr> -rw-r--r-- 1 luke-jr luke-jr 261M Aug 27 09:24 MacOSX.xdelta
02:24 < gmaxwell> there are some other binary patching tools, xdelta was the first that came to mind.
02:25 < sipa> xdelta and xdelta3 seem to be different things
02:28 < gmaxwell> open-vcdiff
02:28 < luke-jr> I'm writing something custom
02:28 < luke-jr> hmm
02:28 < luke-jr> I can't mmap this DMG :<
02:29 < btcdrak> well surely the better solution would be to fix the hfsplus driver?
02:29 < gmaxwell> luke-jr: upgrade to a darn 64 bit OS already luke!
02:31 < luke-jr> oh, if I build with -static I can
02:38 < sipa> i can't get it below 46 MB
02:39 < sipa> that's with 
02:39 < sipa> xdelta3 encode -P $((2**30)) -I 0 -B 2147483648 -W 16777216
02:39 < sipa> (the highest values for each of the limits that work)
02:41 < sipa> it runs too fast, though
02:42 < sipa> this is an 8 GB source file, and it only takes a few seconds
02:42 < gmaxwell> "I demand my np-complete problems solving software be slow"
02:42 < sipa> i demand it actually uses the whole input
02:43 < gmaxwell> try open-vcdiff?
02:44 -!- Ginnarr [~Ginnarr@unaffiliated/ginnarr] has joined #bitcoin-core-dev
02:45 < sipa> 42 MB
02:45 < sipa> gmaxwell: seems that uses the same algorithm as xdelta3
02:46 < gmaxwell> failure?
02:47 < luke-jr> http://codepad.org/Yi5FEWML <-- any obvious bugs or possible optimizations?
02:49 < btcdrak> also, I tried the dmg2img tool, but that doesnt appear to have been updated to handle the altered compression
02:50 < GitHub180> [bitcoin] ajtowns closed pull request #8575: leveldb: generate lib independent of locale sort (master...leveldb-locale-reproducible) https://github.com/bitcoin/bitcoin/pull/8575
02:50 < luke-jr> welp, failed to find the first 82 byte file :/
02:50 < gmaxwell> luke-jr: well the needles may not be in the file in a contigious chunk. 
02:50 < luke-jr> gmaxwell: surely 82 bytes would be :<
02:50 < gmaxwell> unless the file system stores the first n bytes of files in the inode table.
02:51 < luke-jr> hmm
02:51 < gmaxwell> (so that magic is fast)
02:51 < gmaxwell> or just to prevent the indirection
02:51 < luke-jr> would it make sense to target the 2nd 4096 bytes of the file then?
02:53 < gmaxwell> to test your tool grab the second 4096 bytes of the haystack.
02:54 < gmaxwell> but yes, you could try that.
02:54 < luke-jr> http://codepad.org/VvfwXbqo isn't getting anything so far either
02:54 < luke-jr> you mean of the needle, right?
02:55 < gmaxwell> I mean grab a 4096 byte chunk out of the data you are searching in, just to verify your tool works.
02:56 < luke-jr> oh
02:56 < gmaxwell> dd if=5.hfs of=junk bs=4096 skip=1234 count=1
02:56 < luke-jr> "In Mac OS X Snow Leopard 10.6, HFS+ compression was added. In open source and some other areas this is referred to as AppleFSCompression. Compressed data may be stored in either an extended attribute or the resource fork."
02:56 < luke-jr> I bet this is what we're dealing with :|
02:57 -!- fengling [~fengling@58.135.95.136] has quit [Ping timeout: 240 seconds]
02:57 < sipa> seems likely, indeed
02:59 < gmaxwell> bleh
03:00 < luke-jr> http://www.spinics.net/lists/linux-fsdevel/msg55545.html anyone want to implement? :/
03:02 -!- fengling [~fengling@58.135.95.136] has joined #bitcoin-core-dev
03:10 < sipa> leveldb 1.19 was released
03:11 < sipa>  52 files changed, 1976 insertions(+), 429 deletions(-)
03:11 < sipa> (compared to 1.18)
03:11 < gmaxwell> doesn't sound too bad to review.
03:12 < sipa> they added a cache pruning
03:13 < gmaxwell> doesn't sound especially relevant then.
03:13 < sipa> it's been 2 years since their previous release
03:13 < sipa> glad to see activity :)
03:15 < sipa> ARM64 support for memory barriers seems relevant
03:15 < sipa> cache size estimation
03:15 < gmaxwell> oh the arm64 might fix performance on odroid c2
03:28 < sipa> this is interesting but not included: https://github.com/google/leveldb/pull/309
03:29 < gmaxwell> wumpus started benchmarking what that might look like.
03:29 < gmaxwell> odroid c2 has a crc32c accelerator that is sutiable too.
03:35 -!- fengling [~fengling@58.135.95.136] has quit [Ping timeout: 240 seconds]
03:35 -!- harrymm [~wayne@171.5.186.30] has quit [Ping timeout: 264 seconds]
03:47 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
03:51 -!- harrymm [~wayne@49.49.78.120] has joined #bitcoin-core-dev
04:03 < luke-jr> found implementation of HFS+ compression
04:03 < sipa> where?
04:03 < luke-jr> SleuthKit
04:04  * luke-jr suggests we drop manpages from the tarball
04:06 < luke-jr> xz -d <inodes.xz | while read inode filename; do filename="y/$filename"; p=$(dirname $filename); mkdir -vp $p; icat  ../5.hfs $inode >"$filename"; done
04:07  * luke-jr tests result in gitian
04:08 < luke-jr> the tarball is missing a bunch of stuff that looks like dummy files (but still 66 MB)
04:08 < sipa> luke-jr: nice!
04:09 < luke-jr> FWIW, I generated the inode list using mount and stat :p
04:10  * luke-jr would be surprised if SleuthKit didn't have a way to get that info, but didn't see it
04:12 < luke-jr> fls seems to be the tool
04:15 < sipa> sleuthkit added support for HFS+ in version 3.1
04:16 < luke-jr> I have 4.0.2
04:16 < sipa> ubuntu 12.04 has sleuthkit 3.2.3
04:17 < sipa> so i think we're good
04:18 < luke-jr> oh.
04:18 < luke-jr> except gitian doesn't like it :<
04:18 < sipa> due to the missing manpages...?
04:18 < luke-jr> hmm
04:18 < luke-jr> I think symlinks
04:18 < sipa> does it do a checksum check perhaps on the .tgz?
04:18 < luke-jr> can't find the lib for c++
04:18 < luke-jr> it can't, everyone's .tgz is different ;p
04:19 < luke-jr> yeah, I think it's missing symlinks
04:27 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev
04:29 < luke-jr> fls ../5.hfs -rpF 154283 | perl -nle 'm/^(r|l)\S*\s(\d+)\:\s*(.*$)/ && print "$1 $2 $3"' | while read type inode filename; do filename="MacOSX10.11.sdk/$filename"; mkdir -p "$(dirname "$filename")"; if [ "$type" = "l" ]; then ln -s $(icat ../5.hfs $inode) "$filename"; else icat ../5.hfs $inode >"$filename"; fi; done
04:32 < luke-jr> tempting to figure out some kind of trick to download data on demand for this :P
04:39 < luke-jr> hm, surprised there's no single-file curl FUSE fs
04:48 -!- Ginnarr [~Ginnarr@unaffiliated/ginnarr] has quit [Quit: Textual IRC Client: www.textualapp.com]
04:54 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has left #bitcoin-core-dev []
04:54 < luke-jr> gitian build matches
05:09 < sipa> \o/
05:22 < btcdrak> wow
05:26 < luke-jr> ?
05:32 < CodeShark> I think btcdrak just considers perl code to be inherently aesthetically pleasing :p
05:33 < luke-jr> lol
05:33 < luke-jr> latest draft has no perl sadly: http://codepad.org/1wMp5vse
05:33 < luke-jr> I'll wait until I'm actually awake to turn it into a PR
05:34 < luke-jr> -rw-r--r-- 1 luke-jr luke-jr  21M Aug 27 12:33 MacOSX10.11.sdk.tar.gz
05:35 < luke-jr> night
05:38 -!- achow101 [~achow101@206.196.187.145] has joined #bitcoin-core-dev
05:42 < phantomcircuit> luke-jr, the client can take ages to start if you closed it after you downloaded a bunch of blocks but before you processed them
05:42 < phantomcircuit> cause it processes all of them in AppInit2
05:42 < phantomcircuit> (actually it doesn't anymore so this shouldn't be an issue in 0.13.x)
05:43 < sipa> leveldb 1.19 should start up significantly faster
05:44 < phantomcircuit> oh and leveldb reads it's journal which is potentially very slow also
05:52 -!- achow101 [~achow101@206.196.187.145] has quit [Ping timeout: 240 seconds]
05:56 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev
06:00 -!- moli [~molly@unaffiliated/molly] has quit [Read error: Connection reset by peer]
06:05 -!- achow101 [~achow101@206.196.187.145] has joined #bitcoin-core-dev
06:23 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 244 seconds]
06:24 < GitHub3> [bitcoin] sipa opened pull request #8610: Share unused mempool memory with coincache (master...sharemem) https://github.com/bitcoin/bitcoin/pull/8610
06:26 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
06:29 < GitHub62> [bitcoin] sipa opened pull request #8611: Reduce default number of blocks to check at startup (master...fastcheck) https://github.com/bitcoin/bitcoin/pull/8611
06:32 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
06:34 < phantomcircuit> sipa, #8610 instead of doing that can you add a framework for limiting memory globally?
06:35 < GitHub51> [bitcoin] sipa opened pull request #8612: Check for compatibility with download in FindNextBlocksToDownload (master...fixwitban) https://github.com/bitcoin/bitcoin/pull/8612
06:35 < sipa> phantomcircuit: i don't have 5 years
06:35 < phantomcircuit> it can probably even be as simple as a global memory limit goal and percentages for now
06:36 < sipa> percentages isn't good enough
06:36 < sipa> you need something that detects "oh, the mempool actually does not need its maximum usage... let's move some of its allocation elsewhere"
06:36 < sipa> but can change that back when the mempool grows
06:37 < phantomcircuit> percentages and an atomic of how much everything thinks it's using?
06:37 < sipa> what does that solve?
06:37 < phantomcircuit> "usage is below 90% of limit i can exceed my limit"
06:37 < phantomcircuit> i guess you need callbacks to apply memory pressure then
06:37 < sipa> yeah
06:37 < phantomcircuit> but you need that for sharing memory at all
06:38 < sipa> utxo set and mempool are things that are constantly checked anyway
06:38 < sipa> what this PR implements is mempool < maxmempool && coincache + mempool < maxtotal
06:38 < sipa> so arguably it already has something like a global limit (though it's just coincache + mempool)
06:39 < sipa> the mempool has complicated semantics wrt computing relay fee based on its size
06:39 < sipa> making its actual limit dynamic is harder, i think
06:41 < sipa> also, i don't think it's needed to for example give your mempool 10 GB of memory even if you have 64 GB available... that'll just make your memory a sewer for spam that nobody else on the network accepts
07:00 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving]
07:01 < GitHub42> [bitcoin] sipa opened pull request #8613: [preview] LevelDB 1.19 (master...leveldb119) https://github.com/bitcoin/bitcoin/pull/8613
08:00 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has joined #bitcoin-core-dev
08:08 -!- fengling [~fengling@58.135.95.136] has joined #bitcoin-core-dev
08:24 -!- fengling [~fengling@58.135.95.136] has quit [Ping timeout: 240 seconds]
08:39 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has quit [Quit: gtg]
09:03 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has joined #bitcoin-core-dev
09:35 -!- MarcoFalke [~marco@5.199.182.203] has joined #bitcoin-core-dev
09:39 -!- harrymm [~wayne@49.49.78.120] has quit [Ping timeout: 260 seconds]
09:53 -!- harrymm [~wayne@223.207.223.20] has joined #bitcoin-core-dev
10:02 -!- Megaf [~quassel@unaffiliated/megaf] has quit [Read error: Connection reset by peer]
10:08 -!- Megaf [~quassel@unaffiliated/megaf] has joined #bitcoin-core-dev
10:10 -!- achow101 [~achow101@206.196.187.145] has quit [Ping timeout: 244 seconds]
10:22 -!- Megaf [~quassel@unaffiliated/megaf] has quit [Ping timeout: 240 seconds]
10:25 -!- Samdney [~Samdney@dyn-ant666999.hawo.ipv6.uni-erlangen.de] has joined #bitcoin-core-dev
10:36 -!- JackH [~Jack@79-73-189-132.dynamic.dsl.as9105.com] has quit [Ping timeout: 250 seconds]
10:42 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Quit: bye]
10:48 -!- achow101 [~achow101@206.196.187.145] has joined #bitcoin-core-dev
11:19 -!- Lysanders [~Lysanders@unaffiliated/lysanders] has joined #bitcoin-core-dev
11:57 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev
12:11 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has quit [Quit: gtg]
12:15 -!- JackH [~Jack@79-73-189-132.dynamic.dsl.as9105.com] has joined #bitcoin-core-dev
12:19 -!- JackH [~Jack@79-73-189-132.dynamic.dsl.as9105.com] has quit [Read error: Connection reset by peer]
12:22 -!- justanotheruser [~Justan@unaffiliated/justanotheruser] has quit [Disconnected by services]
12:23 -!- justanotheruser [~Justan@24.219.72.136] has joined #bitcoin-core-dev
12:27 -!- justanotheruser [~Justan@24.219.72.136] has quit [Client Quit]
12:28 -!- justanotheruser [~Justan@unaffiliated/justanotheruser] has joined #bitcoin-core-dev
12:45 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has quit [Quit: Leaving]
12:45 -!- justanotheruser [~Justan@unaffiliated/justanotheruser] has quit [Disconnected by services]
12:53 < gmaxwell> sipa: I think we should backport feeler connections. Beyond the security/robustness improvement, they'll help reduce the harm of network density loss w/ segwit.
12:54 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev
13:02 -!- timothy [~quassel@archlinux/trusteduser/DrizztBSD] has quit [Ping timeout: 276 seconds]
13:07 -!- justanotheruser [~Justan@unaffiliated/justanotheruser] has joined #bitcoin-core-dev
13:12 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Quit: laurentmt]
13:18 < GitHub126> [bitcoin] luke-jr opened pull request #8617: Include instructions to extract Mac OS X SDK on Linux using 7zip and SleuthKit (master...gitian_osx_extractor) https://github.com/bitcoin/bitcoin/pull/8617
13:20 < luke-jr> sipa: so have you confirmed already that LevelDB 1.19 is reasonably certain to not have forking changes? or are you assuming we'll do collectively that before merging/releasing?
13:22 < sipa> luke-jr: i'm reasonably certain, yes, but i encourage review
13:23 < sipa> i'm just pring it to bitcoin already to make testing easier
13:53 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev
14:14 -!- achow101 [~achow101@206.196.187.145] has quit [Quit: Leaving]
14:43 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has joined #bitcoin-core-dev
14:59 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has joined #bitcoin-core-dev
15:22 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Read error: Connection reset by peer]
15:26 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
15:27 -!- Lysanders [~Lysanders@unaffiliated/lysanders] has quit [Ping timeout: 258 seconds]
15:27 -!- Alopex [~bitcoin@176.9.70.183] has joined #bitcoin-core-dev
15:32 -!- belcher [~belcher@unaffiliated/belcher] has joined #bitcoin-core-dev
15:47 -!- Yogh [~Yogh@f36186.upc-f.chello.nl] has quit [Ping timeout: 264 seconds]
15:47 -!- MarcoFalke [~marco@5.199.182.203] has quit [Ping timeout: 250 seconds]
15:49 -!- Yogh [~Yogh@f36186.upc-f.chello.nl] has joined #bitcoin-core-dev
15:50 -!- Alopex [~bitcoin@176.9.70.183] has quit [Remote host closed the connection]
15:51 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
15:51 -!- harrymm [~wayne@223.207.223.20] has quit [Remote host closed the connection]
16:14 -!- nibor [~nibor@185.9.34.66] has quit [Ping timeout: 244 seconds]
16:15 -!- nibor [~nibor@185.9.34.66] has joined #bitcoin-core-dev
16:16 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
16:17 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
16:30 -!- belcher [~belcher@unaffiliated/belcher] has quit [Quit: Leaving]
16:43 -!- cryptapus_afk is now known as cryptapus
17:00 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
17:01 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
17:11 -!- belcher [~belcher@unaffiliated/belcher] has joined #bitcoin-core-dev
17:11 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
17:12 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
17:14 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
17:17 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 265 seconds]
17:23 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
17:24 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
17:26 -!- JZA [JZA@gateway/shell/elitebnc/x-wzwcflvrdxfxvjiq] has joined #bitcoin-core-dev
17:38 -!- ghtdak [~ghtdak@unaffiliated/ghtdak] has quit [Quit: WeeChat 1.3]
17:46 -!- belcher [~belcher@unaffiliated/belcher] has quit [Quit: Leaving]
18:01 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
18:02 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
18:37 -!- instagibbs [~instagibb@pool-100-15-118-244.washdc.fios.verizon.net] has quit [Ping timeout: 276 seconds]
18:41 -!- instagibbs [~instagibb@pool-100-15-118-244.washdc.fios.verizon.net] has joined #bitcoin-core-dev
18:48 -!- hybridsole [~hybridsol@unaffiliated/hybridsole] has left #bitcoin-core-dev []
18:55 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-cpnrwcanwjjotelf] has quit [Quit: Connection closed for inactivity]
19:02 -!- juscamarena [~jus@68.237.168.189] has joined #bitcoin-core-dev
19:26 < GitHub95> [bitcoin] rebroad opened pull request #8618: [WIP] Erase orphans only when banned (master...EraseOrphansOnlyWhenBanned) https://github.com/bitcoin/bitcoin/pull/8618
19:27 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-ijxscdtkbvjqwjuw] has quit [Quit: Connection closed for inactivity]
19:34 -!- harrymm [~wayne@49.49.78.160] has joined #bitcoin-core-dev
19:48 -!- harrymm [~wayne@49.49.78.160] has quit [Remote host closed the connection]
19:49 -!- harrymm [~wayne@49.49.78.160] has joined #bitcoin-core-dev
20:01 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has quit [Quit: zz.z.z.z.z.z.z.z.z]
20:11 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has quit [Quit: Leaving.]
20:18 -!- slackircbridge [~slackircb@45.55.41.36] has joined #bitcoin-core-dev
20:19 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
20:20 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
20:21 -!- xiangfu [~xiangfu@58.135.95.141] has quit [Ping timeout: 250 seconds]
20:21 -!- xiangfu [~xiangfu@58.135.95.141] has joined #bitcoin-core-dev
20:21 -!- tucenaber [~tucenaber@unaffiliated/tucenaber] has quit [Ping timeout: 250 seconds]
20:22 -!- wolfspraul [~wolfsprau@bobbin.q-ag.de] has quit [Ping timeout: 250 seconds]
20:22 -!- wolfspraul [~wolfsprau@bobbin.q-ag.de] has joined #bitcoin-core-dev
20:23 < GitHub193> [bitcoin] rebroad closed pull request #8572: [WIP] Don't effectively blacklist pruned nodes. (master...Don'tBanPrunedNodes) https://github.com/bitcoin/bitcoin/pull/8572
20:23 -!- sturles_ [~sturles@ulrik.uio.no] has joined #bitcoin-core-dev
20:23 -!- sturles_ [~sturles@ulrik.uio.no] has quit [Changing host]
20:23 -!- sturles_ [~sturles@unaffiliated/sturles] has joined #bitcoin-core-dev
20:25 -!- jl2012_ [uid133844@gateway/web/irccloud.com/x-wnobatvtknoxtjrn] has joined #bitcoin-core-dev
20:25 -!- tucenaber [~tucenaber@o144.231.lokis.net.pl] has joined #bitcoin-core-dev
20:25 -!- tucenaber [~tucenaber@o144.231.lokis.net.pl] has quit [Changing host]
20:25 -!- tucenaber [~tucenaber@unaffiliated/tucenaber] has joined #bitcoin-core-dev
20:25 -!- cfields [~quassel@unaffiliated/cfields] has joined #bitcoin-core-dev
20:26 -!- jl2012 [uid133844@gateway/web/irccloud.com/x-oifebrzcjgviylpt] has quit [Ping timeout: 258 seconds]
20:26 -!- waxwing [~waxwing@62.205.214.125] has quit [Ping timeout: 258 seconds]
20:26 -!- sturles [~sturles@unaffiliated/sturles] has quit [Ping timeout: 258 seconds]
20:26 -!- Samdney [~Samdney@dyn-ant666999.hawo.ipv6.uni-erlangen.de] has quit [Ping timeout: 258 seconds]
20:26 -!- slackircbridge2 [~slackircb@45.55.41.36] has quit [Ping timeout: 258 seconds]
20:26 -!- cfields_ [~quassel@unaffiliated/cfields] has quit [Ping timeout: 258 seconds]
20:26 -!- jl2012_ is now known as jl2012
20:26 -!- Samdney [~Samdney@dyn-ant666999.hawo.ipv6.uni-erlangen.de] has joined #bitcoin-core-dev
20:26 -!- waxwing [~waxwing@62.205.214.125] has joined #bitcoin-core-dev
20:29 -!- cryptapus is now known as cryptapus_afk
20:37 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
20:38 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
20:49 -!- zooko [~user@73.95.139.43] has joined #bitcoin-core-dev
20:50 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
20:50 -!- Samdney [~Samdney@dyn-ant666999.hawo.ipv6.uni-erlangen.de] has quit [Ping timeout: 258 seconds]
20:51 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
21:06 -!- zooko [~user@73.95.139.43] has quit [Read error: Connection reset by peer]
21:07 -!- zooko [~user@73.95.139.43] has joined #bitcoin-core-dev
21:08 -!- challisto [~challisto@c-76-16-149-33.hsd1.il.comcast.net] has joined #bitcoin-core-dev
21:08 -!- challisto [~challisto@c-76-16-149-33.hsd1.il.comcast.net] has quit [Changing host]
21:08 -!- challisto [~challisto@unaffiliated/challisto] has joined #bitcoin-core-dev
21:13 -!- jgarzik [~jgarzik@104-178-201-106.lightspeed.tukrga.sbcglobal.net] has joined #bitcoin-core-dev
21:13 -!- jgarzik [~jgarzik@104-178-201-106.lightspeed.tukrga.sbcglobal.net] has quit [Changing host]
21:13 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev
21:26 < GitHub129> [bitcoin] rebroad opened pull request #8619: Fix compiler warnings (master...FixCompilerWarnings) https://github.com/bitcoin/bitcoin/pull/8619
21:37 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Quit: bye]
22:08 -!- challisto [~challisto@unaffiliated/challisto] has quit [Quit: Leaving]
22:16 -!- zooko [~user@73.95.139.43] has quit [Ping timeout: 244 seconds]
22:25 -!- MarcoFalke [~marco@5.199.182.203] has joined #bitcoin-core-dev
22:34 -!- MarcoFalke [~marco@5.199.182.203] has quit [Remote host closed the connection]
22:34 -!- MarcoFalke [~marco@5.199.182.203] has joined #bitcoin-core-dev
22:37 < GitHub96> [bitcoin] MarcoFalke closed pull request #8619: Fix compiler warnings (master...FixCompilerWarnings) https://github.com/bitcoin/bitcoin/pull/8619
22:37 -!- fengling [~fengling@58.135.95.136] has joined #bitcoin-core-dev
22:50 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
22:51 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
23:01 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
23:02 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
23:30 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-vzmmpdqitefesuru] has joined #bitcoin-core-dev