--- Day changed Tue Oct 20 2015
00:02 < wumpus> not difficult, probably, a matter of pointing the depends at the NDK toolchain, then the type of slog work that cross-building usually is
00:02 < wumpus> I vaguely remember cfields_ has done it at some point
00:03 < wumpus> midnightmagic: netcraft?
00:06 < midnightmagic> wumpus: Ancient meme. BSD is dying, netcraft confirms it. Used to be the repeated mantra on Slashdot, turn of the century.
00:07 < Luke-Jr> http://bsd.slashdot.org/comments.pl?sid=228247&cid=18495137
00:07 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
00:09 < wumpus> midnightmagic: oh, okay :-) BSD's 'linux desktop' meme
00:11 < wumpus> but it's refreshing for a change compared to the people that complain that 4.8 is too old
00:33 < midnightmagic> good god, I'm so old, smart people don't know what I'm muttering anymore
00:35 < wumpus> heh I know the feeling
01:14 -!- baldur [~baldur@pool-173-52-43-219.nycmny.fios.verizon.net] has quit [Ping timeout: 250 seconds]
01:27 -!- baldur [~baldur@pool-173-52-43-219.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
02:00 < btcdrak> Mempool PR https://github.com/bitcoin/bitcoin/pull/6722 is looking good. lots of ACKs now.
02:06 -!- rubensayshi [~ruben@91.206.81.13] has joined #bitcoin-core-dev
02:20 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev
02:49 < GitHub6> [bitcoin] laanwj opened pull request #6859: http: Restrict maximum size of http + headers (master...2015_10_max_http_headers) https://github.com/bitcoin/bitcoin/pull/6859
03:04 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has joined #bitcoin-core-dev
03:07 < GitHub168> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/da7d57fb9501...488f8517a154
03:07 < GitHub168> bitcoin/master 53b86d0 Daniel Kraft: doc: add comment explaining initial header request...
03:07 < GitHub168> bitcoin/master 488f851 Wladimir J. van der Laan: Merge pull request #6829...
03:07 < GitHub46> [bitcoin] laanwj closed pull request #6829: doc: add comment explaining initial header request (master...doc-getheaders) https://github.com/bitcoin/bitcoin/pull/6829
03:08 < GitHub62> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/488f8517a154...c834f568693b
03:08 < GitHub62> bitcoin/master 7801f43 Eric Lombrozo: Added fPowNoRetargeting field to Consensus::Params that disables nBits recalculation.
03:08 < GitHub62> bitcoin/master c834f56 Wladimir J. van der Laan: Merge pull request #6853...
03:08 < GitHub42> [bitcoin] laanwj closed pull request #6853: Added fPowNoRetargeting field to Consensus::Params (master...fNoRetargeting) https://github.com/bitcoin/bitcoin/pull/6853
03:21 < GitHub88> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/c834f568693b...87e5539e9c50
03:21 < GitHub88> bitcoin/master 0d8b175 MarcoFalke: [rpc-tests] fundrawtransaction: Update fee after minRelayTxFee increase
03:21 < GitHub88> bitcoin/master bd4c22e MarcoFalke: [rpc-tests] Check return code
03:21 < GitHub88> bitcoin/master 87e5539 Wladimir J. van der Laan: Merge pull request #6827...
03:22 < GitHub132> [bitcoin] laanwj closed pull request #6828: [rpc-tests] fundrawtransaction: Update fee after minRelayTxFee increase (master...MarcoFalke-2015-fundrawtransactionTestFix) https://github.com/bitcoin/bitcoin/pull/6828
03:22 < GitHub93> [bitcoin] laanwj closed pull request #6827: [rpc-tests] Check return code (master...MarcoFalke-2015-rpcTestsReturnCode) https://github.com/bitcoin/bitcoin/pull/6827
03:36 < GitHub8> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/87e5539e9c50...ae69a75c554f
03:36 < GitHub8> bitcoin/master e76d9e4 fanquake: [depends] Latest config.guess and config.sub
03:36 < GitHub8> bitcoin/master ae69a75 Wladimir J. van der Laan: Merge pull request #6801...
03:36 < GitHub91> [bitcoin] laanwj closed pull request #6801: [depends] Latest config.guess and config.sub (master...update-config-guess-sub) https://github.com/bitcoin/bitcoin/pull/6801
03:46 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.]
03:54 < GitHub27> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/ae69a75c554f...020c4073a03a
03:54 < GitHub27> bitcoin/master b6d5e32 Alex Morcos: Make fee aware of min relay in pruning.py RPC test
03:54 < GitHub27> bitcoin/master 020c407 Wladimir J. van der Laan: Merge pull request #6841...
03:54 < GitHub193> [bitcoin] laanwj closed pull request #6841: Increase fee in pruning.py RPC test (master...fixPruningRPC) https://github.com/bitcoin/bitcoin/pull/6841
04:30 -!- crescendo [~mozart@unaffiliated/crescendo] has quit [Remote host closed the connection]
04:36 < GitHub43> [bitcoin] laanwj pushed 1 new commit to 0.11: https://github.com/bitcoin/bitcoin/commit/072032448be5263f58cbd6b47f61edc7bb8210e1
04:36 < GitHub43> bitcoin/0.11 0720324 Alex Morcos: Make fee aware of min relay in pruning.py RPC test...
04:50 < GitHub65> [bitcoin] MarcoFalke opened pull request #6860: Drop minRelayTxFee to 1000 (master...MarcoFalke-2015-minRelayTxFeeDrop) https://github.com/bitcoin/bitcoin/pull/6860
05:00 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev
05:08 -!- ParadoxSpiral [~ParadoxSp@p508B9014.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
05:25 < GitHub23> [bitcoin] laanwj closed pull request #6860: Drop minRelayTxFee to 1000 (master...MarcoFalke-2015-minRelayTxFeeDrop) https://github.com/bitcoin/bitcoin/pull/6860
05:58 < wumpus> jgarzik: you can't be SERIOUS that truncating a 64 bit number to a 32 bit number was a good idea!??!  https://github.com/bitcoin/bitcoin/issues/6765#event-440267158
05:59 < jgarzik> wumpus, You are confusing an issue with a PR
05:59 < wumpus> this is completely crazy
05:59 < jgarzik> wumpus, calm down please.  The issue is univalve conversion cause unintended behavior changes that are as yet not documented or audited
05:59 < wumpus> it is simply not a bug
06:00 < jgarzik> wumpus, objecting to a solution that was not proposed as a PR is fine
06:00 < wumpus> EWONTFIX
06:00 < jgarzik> wumpus, it is a user noticeable behavior change that potentially breaks people in the field
06:00 < jgarzik> wumpus, who knows what else breakage was caused, until analysis is complete
06:00 < wumpus> if you want to change documentation, be my guest, I don't think there is any documentation that says yo ucan provide numbers there that exceed 32 bit numeric range
06:00 < wumpus> well at most you'll get an error now
06:00 < wumpus> instead of dumbly ANDing off the upper bits
06:01 < wumpus> which can cause much worse problems, ask the openssl devs
06:01 < jgarzik> wumpus, not disagreeing, please read what I wrote
06:01 < jgarzik> wumpus, user breakage - breakage was noticed - let's be methodical
06:01 < wumpus> well, do you intend on doing analysys thee?
06:01 < wumpus> if not, I want to close the issue and move on
06:02 < jgarzik> wumpus, yes the issue should stay open until this is analyzed
06:02 < wumpus> I disagree that it is so important
06:02 < jgarzik> wumpus, without analysis it is provably impossible to say what is the impact
06:02 < wumpus> but anyhow, if you look into this this week, I'm fine with keeping it open
06:04 < wumpus> I don't think you'll find anyone that thinks the new behavior doesn't make sense
06:05 < wumpus> I can ask a few people if you think jonasschnelli, dcousens and mine opinion isn't enough
06:06 < wumpus> arguably if you were using large values in the field it was already broken. You *thought* you were parsing a large value, but you could have been passing a negative value for all you know
06:06 < jonasschnelli> i think the out of range exception behavior is okay and expected
06:08 < jonasschnelli> But we need to write some univalue release notes anyways and could mention this there
06:09 < wumpus> 1000000000000 turns into -727379968, for example
06:10 < jgarzik> wumpus, Please consider the wider picture.  You are focusing on one impacted callsite - and I consistently agree with you that the new behavior at that callsite is better - the issue is that that is not the only .get_int() in the entire codebase.  There may be other user visible breakage where the impact is not so fortunate.  Do not blindly assume.
06:10 < wumpus> on other  callsites the breakage would be exactly the same
06:10 < jgarzik> you hope and assume...
06:10 < wumpus> not, I'm sure
06:10 < wumpus> get_int() simply does a range check now
06:10 < wumpus> that is safer
06:11 < wumpus> if you  have one example where this causes dangerous behavior, just say so
06:11 < wumpus> if not, please stop this nonsense
06:11 < jonasschnelli> release notes -> mention integers will now face a range check ->  done?
06:11 < jgarzik> the new behavior is throwing an exception versus returning a range bound value without an exception
06:11 < wumpus> the old behavior was dangerous, not the newo ne
06:12 < jgarzik> throwing an exception causes different error return messages and different downstream behaviors
06:12 < jgarzik> which in turn cause user visible interface changes
06:12 < jonasschnelli> jgarzik: i agree: it's and API change. But in my opinion one that is worth to take.
06:12 < wumpus> but ony if the input *was already invalid*
06:12 < jgarzik> jonasschnelli, yes
06:12 < jgarzik> jonasschnelli, and my point has always been it needs to be analyzed and documented
06:12 < wumpus> at least now you are told that the input is invalid, instead of replacing it with an effectively random number
06:13 < jgarzik> not simply make the assumption it is OK
06:13 < jgarzik> that is extremely poor software engineering - hope and pray
06:13 < wumpus> well then do something better
06:13 < jgarzik> keeping an issue open until it is analyzed fully is not a huge burden
06:13 < wumpus> opening lots of issues that no one ever looks at is just doesn't cut it either
06:14 < wumpus> it is a burdern, though
06:14 < jonasschnelli> Mention it in the release notes is okay, but not mandatory IMO. If a API consumer was relying range bound value he needs to be slapped anyways. :)
06:14 < jgarzik> don't make mountains out of molehills.
06:14 < wumpus> well possibly they didn't even know the range was not 64 bits. This bug could even be exploitable in some cases
06:14 < wumpus> at least now they aretold
06:15 < jgarzik> the issue is assigned to me now.  please stop hyperventilating over a minor issue.
06:15 < wumpus> there could be cases where *application side* range checks can be bypassed by providing large numbers, unaware that bitcoind cuts them off
06:36 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
06:45 -!- helo_ is now known as helo
06:52 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.]
06:58 < GitHub186> [bitcoin] laanwj closed pull request #6855: [REST] API versioning (master...restVersioning) https://github.com/bitcoin/bitcoin/pull/6855
07:11 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: Textual IRC Client: www.textualapp.com]
07:15 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
07:20 < morcos> wumpus: I'm a bit out of my depth in the memory "leak" related to getblocktemplate,  but i have an idea on the cause
07:20 < morcos> in CreateNewBlock, we have a vecPriority that is approximately equal to the size of the mempool.
07:21 < morcos> the first time you call getblocktemplate your resident usage jumps by the size of your mempool
07:21 < morcos> repeated calls to getblocktemplate will continue to cause that to happen, up to the number of rpc threads you are running
07:22 < morcos> this is still mostly guesswork at this point, but its not clear to me why that memory used by vecPriority appears to be sticking around (a copy per thread)
07:29 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.]
07:30 < wumpus> morcos: strange; I don't think we use any thread-local storage at all
07:31 < morcos> well, like i said, still guess work, could be wrong.  
07:31 < morcos> but vecPriority is a local variable, so that would be thread-local storage right, it just should go away when the function exits
07:32 < morcos> but maybe its some stl "optimization" to keep it allocated or something
07:33 < jgarzik> In some thread systems (not sure about RPC threads), the threads do not die - they are recycled
07:34 < jgarzik> so a huge stack would remain
07:34 < morcos> but vecPriority is local to the function CreateNewBlock
07:34 < jgarzik> indeed, that should go out of scope
07:35 < jgarzik> as should CCoinsViewCache
07:36 < wumpus> the threads certainly stay around
07:36 < jgarzik> also in terms of allocator behavior, large amounts of memory are not necessarily reclaimed at the process level.  In the C library (not sure about C++), lots of free(3) memory is recycled back into allocator, not returned to system via munmap(2)
07:36 < wumpus> but the stack can't grow that far, a vector is allocated on the heap not the stack
07:37 < wumpus> (also pthreads default stack size is 8mb per thread; hardly possible to cause so much 'leaking' that way)
07:37 < jgarzik> So even though the app has released memory to the allocator, the allocator will not necessarily release memory to the system, as shown in top(1)
07:38 < wumpus> right, freed memory is not returned to the system immediately
07:38 < wumpus> sometimes never, depending on the size of the allocation
07:38 < jgarzik> Process virtual memory size therefore just sits at its maximum, even if unused by app
07:38 < jgarzik> *sometimes
07:39 < jgarzik> (recent libraries do release memory if chunks are large enough)
07:39 < wumpus> (but it's reused if the application allocates again, so it should not result in growing process size over time in itself)
07:39 < morcos> we're talking about resident memory, not virtual memory, but you're saying it could still reflect there
07:39 < jgarzik> nod
07:39 < wumpus> do multiple getblocktemplates run at the same time indifferent threads? that'd explain it 
07:39 < morcos> wumpus: that's why i put "leak" in quotes, it seems entirely possible to me that the memory usage is limited to an excess of #rpcthreads * maxseenmempoolsize
07:40 < jgarzik> per-thread heaps are certainly possible in C++
07:40 < jgarzik> does memory exceed RPC thread count
07:40 < morcos> wumpus: i was running them serially
07:40 < jgarzik> seems like the steady state could be reached once all RPC threads are "huge" :)
07:40 < wumpus> morcos: ok 
07:41 < morcos> yeah, so there might be a steady state, but the steady state is particularly inefficient if you have a lot of RPC threads
07:41 < wumpus>  the behavior seems consistent with per-thread heap, yes
07:41 < wumpus> default rpcthreads is only 4, there shouldn't usually be a reason to increase that
07:42 < wumpus> (especially as everything holds the cs_main lock anyway, so there is only very little scope for paralellism)
07:42 < morcos> yes but 4 * max_mempool_usage is a lot of extra memory right?
07:42 < jgarzik> not if you have a studly machine
07:42  * jgarzik runs
07:43 < wumpus> would be interesting to use google heap profiler to see if the memory gets released in between the calls
07:43 < wumpus> if it doesn't show up in there, it's the matter of memory not being released to the OS
07:44 < jgarzik> Amusingly I bet you could actually go faster with fork(2) and a lockless COW mempool copy that disappears upon child exit(2).    (unrealistic due to portability... not a real proposal)
07:45 < jgarzik> faster and less memory bloat :)
07:45 < wumpus> (using heap profiler is easy as   LD_PRELOAD="/.../libtcmalloc.so"  src/bitcoind , see https://gperftools.googlecode.com/svn/trunk/doc/heapprofile.html   It can also track mmaps but I had less success using it for that)
07:46 < wumpus> well if that means forking a child for every http connection, I don't think it'd be faster, but yeah you could use the processes a few times and then have them exit...
07:49 < wumpus> I'm fairly sure apache does as much
07:49 < jgarzik> nah just fork off for that one RPC call, and have the thread wait for results
07:50 < jgarzik> when you're making a complete copy of the process you can do stuff like that :)
07:50 < jgarzik> but doesn't work at all on Windows so...
07:50 < jgarzik> on Windows, you have to fake it by executing your own .exe again
07:51 < jgarzik> And correct - Apache re-uses both processes and threads for a limited time, then closes then - for reasons precisely like the ones we are seeing
07:51 < jgarzik> limited recycle prevents memory from building up
07:51 < jgarzik> *closes them
07:51 -!- rubensayshi [~ruben@91.206.81.13] has quit [Remote host closed the connection]
07:53 < wumpus> morcos: can you try export MALLOC_ARENA_MAX=1 , this is supposed to disable per-thread arenas for newer glibc
07:54 < morcos> sure i will try
07:54 < wumpus> (from  https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en  they see similar excessive allocation behavior)
07:57 < wumpus> by default it creates an arena for every CPU core
08:01 < jgarzik> yep - that's so thread local structures can run lock free simply by accessing arena_vector[my_cpu_core_number]
08:02 < wumpus> from a performance point of view it makes a lot of sense
08:02 < wumpus> at least when there is actual paralellism :)
08:04 < jgarzik> The cs_main lock has amusing historical parallels.  both freebsd and linux kernels had a Big Kernel Lock - a single lock that nearly everything would grab - during the early days of their conversion to SMP/parallel kernels
08:04 < wumpus> yes and don't forget python's big lock :)
08:04 < jgarzik> that too
08:05 < jgarzik> python is so bad at threading you -have- to fork ;p
08:06 < wumpus> but I hope bitcoin's is more like the BSD/Linux kernel lock than the python one, at least the former got rid of it at some point
08:06 < GitHub73> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/020c4073a03a...e26a3f6713f8
08:06 < GitHub73> bitcoin/master f3525e2 Jorge Timón: Chainparams: Replace CBaseChainParams::Network enum with string constants (suggested by Wladimir)
08:06 < GitHub73> bitcoin/master 55a8975 Jorge Timón: Chainparams: Translations: DRY: options and error strings...
08:06 < GitHub73> bitcoin/master e26a3f6 Wladimir J. van der Laan: Merge pull request #6235...
08:06 < GitHub151> [bitcoin] laanwj closed pull request #6235: Chainparams: Translations: DRY: options and error strings (master...chainparams-dry) https://github.com/bitcoin/bitcoin/pull/6235
08:07 < wumpus> with python it's such an essential part of the design that it's unrealistic to get rid of it. At least that was in the 2.x days, i don't know about 3.x
08:09 < GitHub87> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/e26a3f6713f8...c6de5cc88614
08:09 < GitHub87> bitcoin/master e253e83 Matt Corallo: Update debian/changelog and slight tweak to debian/control
08:09 < GitHub87> bitcoin/master c7b36cc Matt Corallo: Change URLs to https in debian/control
08:09 < GitHub87> bitcoin/master c6de5cc Wladimir J. van der Laan: Merge pull request #6796...
08:09 < GitHub160> [bitcoin] laanwj closed pull request #6796: Update debian/changelog and slight tweak to debian/control (master...debian) https://github.com/bitcoin/bitcoin/pull/6796
08:10 < jgarzik> yep
08:10 < jgarzik> and we're well on our way to peeling off the cs_main (bsd/linux road)
08:10 < jgarzik> imo
08:31 < morcos> wumpus: sorry its taking me a while, i was trying to get a full mempool but not also spam the rest of the network...  i haven't run the getblock templates yet, but it sure does make bitciond use a lot less memory in the first place (using MALLOC_ARENA_MAX=1)
08:32 < morcos> normally i'm using close to 1G shortly after starting, with this, it was about 250M
08:38 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev
08:46 < wumpus> morcos: good to know! thanks for testing, I always had this issue where only very little memory usage would show up in the heap profiler, but top showed much more, now we understand why! 
08:48 < wumpus> and the more cores, the more it will use
08:48 < wumpus> should add that one to the use-less-memory tips
08:49 < morcos> wumpus: yep, after repeated calls to getblocktemplate, it only jumped in memory usage once
08:50 < morcos> i think concentrating on vecPriority was misguided (its not actually that big, just ptrs to txs) but between all the stuff thats stl-allocated in CNB, that probably explains the big jump.  i havent' gone through really to see if it adds up yet
08:50 < morcos> it does still seem like a big jump.
08:50 < morcos> whether it happens 1x or #rpcthreads times
08:51 < morcos> i suppose it might depend on how much the CCoinsViewCache allocates?
08:52 < wumpus> that depends; they will cache everything that is queried through them
08:55 < jgarzik> wumpus, any remaining merge blockers for #6722 in your view?
08:58 < morcos> +1 for merging it sooner rather than later. only reason not to merge i think is if we can coerce more people into review/test.  but beyond that, lets get it out into the wild to find more bugs.
08:58 < morcos> however its worth considering merging the lower chain limits soon as well
08:58 < jgarzik> yeah this sort of stuff is perfect for master
09:00 < GitHub85> [bitcoin] jtimon reopened pull request #6625: BLOCKING: Consensus: Move blocksize and related parameters to consensusparams ...without removing consensus/consensus.h [#6526 alternative] (master...consensus-blocksize-0.12.99) https://github.com/bitcoin/bitcoin/pull/6625
09:01 < jgarzik> the Internet is always a better test lab than anything we cook up in private
09:01 < jgarzik> merge easy, revert easy
09:01 < jgarzik> I would have merged it myself if it were a normal change, given the ACK quotient, but for a change of this notability I want wumpus, gmaxwell, sipa etc. approval of some sort (sipa has already ACK'd FTR)
09:03 < wumpus> voila https://gist.github.com/laanwj/efe29c7661ce9b6620a7#linux-specific
09:03 < wumpus> jgarzik: will  take a loook
09:03 < jgarzik> I've been testing it as well
09:04 < jgarzik> wumpus, +1 gist
09:06 < GitHub107> [bitcoin] jtimon closed pull request #6672: Consensus: Separate most consensus functions to consensus.cpp (master...consensus-moveonly-0.12.99) https://github.com/bitcoin/bitcoin/pull/6672
09:10 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev
09:10 < morcos> wumpus: do you think thats a sufficient "fix" for the memory usage.   
09:11 < morcos> still not clear exactly how much heap allocation is happening in getblocktemplate
09:11 < morcos> but lets say for the sake of argument its on the order of your mempool usage size
09:11 < morcos> now you're saying every node out there (that doesn't run with your low mem tip) is going to use 5x maxmempoolsize
09:12 < morcos> at least
09:12 < morcos> i guess only if they run getblocktemplate
09:13 < morcos> but maybe we can do something to try to reduce that.  we know for instance that there is no need for this particular memory usage to worry about trying to allocate at the same time
09:13 < morcos> if that was even a static variable that was just cleared between calls, it would persist the memory all the time, but only 1x right?
09:14 < morcos> so maybe vecPriority and view and whatevers using memory in createnewblock
09:18 < GitHub92> [bitcoin] sdaftuar closed pull request #6557: Mempool limiting with descendant package tracking (master...mempool-packages) https://github.com/bitcoin/bitcoin/pull/6557
09:19 < gmaxwell> memory usage I'm seeing is certantly beyond 5x.
09:20 < gmaxwell> 3.77GB res compared to   "usage": 90103744
09:26 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
09:29 < morcos> gmaxwell: i think it turns out this arena per core thing will affect a lot of bitcoind mem usage...   also, i think its the max ever mempool usage that matters, not current
09:31 < gmaxwell> ah! reading more backscroll, so the idea is that the pre-thread (or whatever) arena is ending up allocating but not sbrking data in each thread (or core?) that runs a GBT and thus has a high peak usage because it's copied the mempool?
09:33 < gmaxwell> Having control of stuff like this is why firefox uses jemalloc
09:53 < morcos> gmaxwell: yeah to the limit of my understanding.  except still not exactly clear what is taking up the memory, b/c the mempool isn't copied.  but could be the CoinsViewCache.
10:26 -!- CodeShark [CodeShark@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
10:29 < wumpus> coinsviewcache has methods to measure the size
10:38 < gmaxwell> so, reading the page about https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en .. it looks like thats about about virtual usage, not RES.
10:39 < gmaxwell> (not that it wouldn't also be good to reduce virtual usage, at least on 32bit.. but I'm seeing about 4x resident than what I expect)
10:41 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
10:45 < morcos> gmaxwell: yeah that confused me as well.  but from my very ad hoc testing it seemed to affect RES
10:51 < wumpus> gmaxwell: well the memory is at least temporary resident when it is used
10:53 < wumpus> it's not just reserving an arena, it's actually returning memory from it, to be used and later freed
10:53 < wumpus> after which it sticks around, no longer touched so it doesn't have to stay resident, but that depends on mem pressure
10:54 < wumpus> (also if you keep sending getblocktemplate it probably reuses them in more-or-less round-robin fashion)
10:56 < gmaxwell> oh i bet that interacts super well with people who've cranked rpc threads to get around the issues with rpcblocking.
11:04 < wumpus> nah the number of threads is not that relevant, just the number of cores
11:05 < wumpus> it tries to balance arenas over cores
11:05 < wumpus> (which makes a lot of sense for performance, but in our case it's not useful)
11:05 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 272 seconds]
11:06 < gmaxwell> I wonder if its ever useful for large allocations. hm. then again the usage is probably made out of zillions of tiny allocations.
11:07 < wumpus> we had a workstation with a crazy NUMA motherboard at uni, where each processor had its own memory. I guess that's the only case where it makes sense for large, infrequent allocations.
11:08 < wumpus> (but yes in bitcoin we use tons of small allocations)
11:16 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
11:26 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
11:29 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Read error: Connection reset by peer]
11:32 -!- wangchun [~wangchun@li414-193.members.linode.com] has joined #bitcoin-core-dev
11:34 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
11:43 < CodeShark> wumpus: it seems test_bitcoin behavior has changed recently. not too long ago, if I ran src/test/test_bitcoin --run_tests=<testname> from the root project directory, it would only run that specific test - but now it seems to run all of them
11:44 < CodeShark> am I tripping?
11:52 < wumpus> --run-tests= takes a test suite name, not an individual test name
11:53 < CodeShark> I mean singular
11:53 < CodeShark> --run_test=
11:53 < wumpus> if the test suite doesn't exist, it somehow gets them all
11:53 < wumpus> that's not possible
11:53 < CodeShark> let me verify what I'm saying to make sure it's accurate...
11:53 < wumpus> yes it's singular run-test, but it takes a suite name
11:54 < CodeShark> if the test suite doesn't exist I get an error
11:54  * wumpus test_bitcoin -l test_suite   will show exactly what test suites and tests it's executing, and how long they take
11:55 < wumpus> that behavior is part of boost test, so if it changed, you probably changed your boost version
11:56 < wumpus> -run-tests= isn't picked up at all here, -t is
11:57 < CodeShark> travis is apparently not liking alert_tests on windows https://travis-ci.org/bitcoin/bitcoin/jobs/86404141
11:58 < wumpus> seems to be passing fine here?
11:58 < CodeShark> I don't get that error at all - I didn't touch alert_tests
11:58 < CodeShark> I mean, I don't understand that error
12:01 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev
12:02 < CodeShark> how can I try to reproduce that error locally?
12:02 < wumpus> I had a quite crazy test error in OpenBSD with Alert_tests today: http://s28.postimg.org/duxcxvrh9/Schermafdruk_van_2015_10_20_10_08_52.png  (no, probably not related, something is very wrong there)
12:02 < wumpus> you can reproduce it in travis?
12:03 < CodeShark> I'm not very familiar with travis - can I get it to run the test again remotely?
12:03 < CodeShark> do I need to force push?
12:03 < wumpus> I can do that, let me see
12:08 < wumpus> should be running again now
12:12 < CodeShark> thanks, wumpus - I tried doing a force push anyhow
12:12 < wumpus> oh then we triggered it twice, that probably just slows it down
12:14 < jgarzik> RE malloc arenas,
12:14 < jgarzik> http://journal.siddhesh.in/posts/malloc-per-thread-arenas-in-glibc.html
12:14 < jgarzik> (apologies if dup; didn't see it in scrollback)
12:16  * jgarzik looks for some way to cap the number of arenas besides MALLOC_ARENA_MAX
12:27 < wumpus> don't think I saw that one yet
12:28 < Luke-Jr> heh, when I wanted to run a specific test, I edited the makefile to not include the rest <.<
12:29 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Ping timeout: 240 seconds]
12:29 < wumpus> "Freed blocks near the end of heaps in an arena are given back to the system either by using madvise(MADV_DONTNEED) or by explicitly unmapping." that doesn't seem to happen for us, probably due to heap fragmentation
12:41 < jgarzik> nod - lots of little allocations will do that :/
13:02 < gmaxwell> Change to jemalloc which should also reduce fragmentation.
13:02 < gmaxwell> :)
13:10 -!- CodeShark [CodeShark@cpe-76-167-237-202.san.res.rr.com] has quit [Ping timeout: 264 seconds]
13:11 -!- phantomcircuit [~phantomci@strateman.ninja] has joined #bitcoin-core-dev
13:19 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Quit: Leaving]
13:41 -!- ParadoxSpiral [~ParadoxSp@p508B9014.dip0.t-ipconnect.de] has quit [Remote host closed the connection]
13:46 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has quit [Ping timeout: 264 seconds]
13:47 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving]
14:06 -!- CodeShark [~CodeShark@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
14:16 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.]
16:42 -!- CodeShark [~CodeShark@cpe-76-167-237-202.san.res.rr.com] has quit [Ping timeout: 272 seconds]
17:36 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.]
18:14 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-upqwkcyzshifosnk] has quit [Quit: Connection closed for inactivity]
19:53 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
19:54 -!- CodeShark_ [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
19:54 -!- berndj [~berndj@azna.co.za] has quit [Quit: ZNC - http://znc.in]
19:55 -!- berndj [~berndj@azna.co.za] has joined #bitcoin-core-dev
19:55 -!- berndj is now known as Guest60457
19:55 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Client Quit]
19:55 -!- CodeShark_ [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Client Quit]
19:56 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
19:56 -!- Guest60457 [~berndj@azna.co.za] has quit [Max SendQ exceeded]
19:57 -!- berndj-blackout [~berndj@azna.co.za] has joined #bitcoin-core-dev
19:57 -!- berndj-blackout [~berndj@azna.co.za] has quit [Max SendQ exceeded]
19:57 -!- berndj-blackout [~berndj@azna.co.za] has joined #bitcoin-core-dev
19:58 -!- berndj-blackout [~berndj@azna.co.za] has quit [Max SendQ exceeded]
19:58 -!- berndj-blackout [~berndj@azna.co.za] has joined #bitcoin-core-dev
20:00 -!- berndj-blackout [~berndj@azna.co.za] has quit [Max SendQ exceeded]
20:01 -!- berndj-blackout [~berndj@azna.co.za] has joined #bitcoin-core-dev
20:02 -!- berndj-blackout [~berndj@azna.co.za] has quit [Max SendQ exceeded]
20:03 -!- berndj-blackout [~berndj@azna.co.za] has joined #bitcoin-core-dev
20:07 -!- berndj-blackout [~berndj@azna.co.za] has quit [Max SendQ exceeded]
20:08 -!- berndj-blackout [~berndj@azna.co.za] has joined #bitcoin-core-dev
20:09 -!- berndj-blackout [~berndj@azna.co.za] has quit [Max SendQ exceeded]
20:29 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Read error: Connection reset by peer]
20:29 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
21:31 -!- CodeShark_ [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
21:36 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Ping timeout: 256 seconds]
22:01 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
22:10 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
23:16 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-zpbkhnefbiobmuzl] has joined #bitcoin-core-dev
23:21 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.]
23:42 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has quit [Ping timeout: 250 seconds]
23:47 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has joined #bitcoin-core-dev
23:50 < GitHub9> [bitcoin] laanwj pushed 14 new commits to master: https://github.com/bitcoin/bitcoin/compare/c6de5cc88614...3b20e239c602
23:50 < GitHub9> bitcoin/master 78b82f4 Suhas Daftuar: Reverse the sort on the mempool's feerate index
23:50 < GitHub9> bitcoin/master 49b6fd5 Pieter Wuille: Add Mempool Expire function to remove old transactions...
23:50 < GitHub9> bitcoin/master 9c9b66f Matt Corallo: Fix calling mempool directly, instead of pool, in ATMP
23:50 < GitHub173> [bitcoin] laanwj closed pull request #6722: Limit mempool by throwing away the cheapest txn and setting min relay fee to it (master...mempoollimit) https://github.com/bitcoin/bitcoin/pull/6722
23:50 < BlueMatt> heyyyyyyyy
23:57 < jonasschnelli> Nice
23:58 < jonasschnelli> Finally