--- Day changed Fri Dec 02 2016 00:00 -!- xiangfu [~xiangfu@223.223.187.142] has quit [Ping timeout: 260 seconds] 00:00 < wumpus> luke-jr: will have a look in a moment 00:01 -!- xiangfu [~xiangfu@223.223.187.142] has joined #bitcoin-core-dev 00:02 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Remote host closed the connection] 00:03 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 00:07 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Ping timeout: 244 seconds] 00:13 < BlueMatt> dcousens: and the "add fee" logic will continue to be maintained....but that isnt the "priority" code - this refers specifically to coin days-destroyed logic 00:15 < luke-jr> the priority code will be as well. 00:15 < sipa> no, it won't 00:15 < wumpus> luke-jr: looks ok to me - though I'm not entirely sure about the qt memory leak commits, they are all pretty minor one-time leaks, so users shouldn't notice it 00:15 < sipa> it serves no function anymore 00:15 < luke-jr> sipa: yes it does, the same function it always served: anti-spam 00:15 < wumpus> luke-jr: still makes sense to clean them up, but not sure whether the backport is risky 00:16 < luke-jr> wumpus: I don't care strongly if you want to skip those 00:16 < luke-jr> the backport may be risky, but if we have an rc or two, I'd put it in 00:16 < wumpus> luke-jr: also the menu reparenting may have subtle behavior issues I hadn't considered (though none reported yet) 00:16 < wumpus> yes, sure 00:16 < sipa> luke-jr: so blocks with 950 kb of spam is fine, and 50kb of transactions from bitcoin old timers that doesn't really pays miners a competitive price will save anything? 00:17 < sipa> the only scalable way to deal with this is economic incentives 00:17 < luke-jr> sipa: it would be better than 1000 kb of spam :P 00:17 < wumpus> lukanyhow I'll pull them into #9264 00:17 < sipa> and face it, that is how it works in practice now 00:17 < gribble> https://github.com/bitcoin/bitcoin/issues/9264 | 0.13.2 backports by laanwj · Pull Request #9264 · bitcoin/bitcoin · GitHub 00:17 < luke-jr> if we go to exclusively economic incentives, we'd need to remove ALL spam filtering.. 00:18 < sipa> luke-jr: there are still local node dos protections 00:18 < sipa> which everyone has an incentive to maintain 00:18 < sipa> and don't requires miners to be gatekeepers of good and bad transactions 00:19 < luke-jr> sipa: if we had a system that wasn't suffering from spam, removing priority might make sense. but we don't. 00:19 < sipa> luke-jr: removing priority will have 0 impact 00:21 < sipa> wallets don't use it anymore, (almost all) miners don't use it anymore - even if it was usable as a means to distinguish better from worse usage of block space, it isn't anymore 00:21 < luke-jr> has someone shown that to be true? 00:22 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 00:22 < luke-jr> last time I looked, a large % of transactions in blocks were in the priority area 00:22 < luke-jr> (not majority-large, but not <5% either) 00:22 < sipa> fair enough, i have no actual data on his 00:22 < sipa> *this 00:23 < sipa> but how do you measure that? 00:23 < luke-jr> I wrote a RPC call that analyzed the sort order 00:27 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev 00:27 * luke-jr tries porting it to 0.13 00:28 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Client Quit] 00:40 -!- arowser_ [~quassel@106.120.101.38] has quit [Ping timeout: 250 seconds] 00:40 -!- arowser [~quassel@106.120.101.38] has joined #bitcoin-core-dev 00:46 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has joined #bitcoin-core-dev 00:53 -!- jannes [~jannes@178.132.211.90] has quit [Quit: Leaving] 00:54 < luke-jr> ugh this thing is slooooooow 01:00 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 01:00 -!- abpa [~abpa@2602:306:b837:dbf0:10d2:337:7ea0:6306] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 01:01 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds] 01:07 < jtimon> for those interested in more configurable testchains: https://github.com/bitcoin/bitcoin/pull/8994#issuecomment-264406053 01:17 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has quit [Ping timeout: 250 seconds] 01:17 -!- laurentmt [~Thunderbi@80.215.178.54] has joined #bitcoin-core-dev 01:29 -!- juscamarena [~justin@47.148.176.74] has joined #bitcoin-core-dev 01:35 -!- e4xit_ [~e4xit@cpc1-cmbg20-2-0-cust188.5-4.cable.virginm.net] has joined #bitcoin-core-dev 01:37 -!- e4xit [~e4xit@cpc1-cmbg20-2-0-cust188.5-4.cable.virginm.net] has quit [Ping timeout: 246 seconds] 01:37 -!- e4xit_ is now known as e4xit 01:50 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Quit: bye] 02:02 < luke-jr> oh, it'd also imply removing the soft blockmaxsize, which is essential since >1 MB blocks are not safe right now. which would therefore imply segwit should be rejected. not a good hole to go down IMO. 02:23 -!- cannon-c [6cab83bc@gateway/web/freenode/ip.108.171.131.188] has joined #bitcoin-core-dev 02:25 -!- laurentmt1 [~Thunderbi@80.215.234.136] has joined #bitcoin-core-dev 02:26 -!- laurentmt [~Thunderbi@80.215.178.54] has quit [Ping timeout: 268 seconds] 02:26 -!- laurentmt1 is now known as laurentmt 02:35 -!- xiangfu [~xiangfu@223.223.187.142] has quit [Ping timeout: 250 seconds] 02:35 -!- xiangfu [~xiangfu@223.223.187.142] has joined #bitcoin-core-dev 02:39 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 02:42 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 268 seconds] 03:02 -!- cannon-c [6cab83bc@gateway/web/freenode/ip.108.171.131.188] has quit [Quit: Page closed] 03:09 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev 03:14 -!- JackH [~laptop@79-73-189-171.dynamic.dsl.as9105.com] has joined #bitcoin-core-dev 03:17 -!- laurentmt [~Thunderbi@80.215.234.136] has quit [Quit: laurentmt] 03:21 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has joined #bitcoin-core-dev 04:00 < luke-jr> sipa: CPFP and some other weirdness I don't recognise kinda made my analyzer useless :/ 04:22 -!- BashCo_ [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 04:22 < luke-jr> jonasschnelli: is it just me, or is 0a261b63fd4f1b07431f8a65762ef9f1ef1c11c8 a bugfix that should be backported? 04:25 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Ping timeout: 248 seconds] 04:37 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has quit [Ping timeout: 268 seconds] 04:52 < gmaxwell> luke-jr: I think we should remove the softmax blocksize. I think it's harmful to the network to have some miners producing smaller blocks when there is a high fee backlog. 04:56 < luke-jr> so just ignore all the much bigger problems large blocks are causing? why is fee so important? it doesn't seem likely larger blocks are going to significantly change the fee rate either.. :/ 05:00 < gmaxwell> luke-jr: having it set lower makes the larger block problem _worse_: vitually all miners just set it to the maximum, and then we pretend like we've done something useful there instead of actually doing something useful. (like implementing a dynamic soft cap so that when demand goes down feerates don't drob absurdly low) 05:00 < gmaxwell> s/drob/drop/ 05:01 < luke-jr> not sure I follow; that sounds like the topic has changed to the *default* maxblocksize, but I'm talking about the option itself. 05:01 < luke-jr> wouldn't the latter just be the min fee setting? 05:05 < gmaxwell> luke-jr: sort of. minfee setting is really a relay setting, it gets in the way of CPFP. Should be intentionally low 05:07 < luke-jr> perhaps the gradual min-feerate would be a good thing to reintroduce 05:07 < gmaxwell> I think the problems of propagation are more or less solved for the moment, so the concerns are bulk blockchain growth and fee behavior predictablity. Having a couple miners going around confirming stuff with absurdly low fees or failing to confirm stuff with decent fees doesn't help. 05:08 < luke-jr> hmm, would gradual min-feerate break prediction? 05:09 < gmaxwell> No. At least if constructed right it should improve it (esp if the prediction is aware of it). 05:10 < morcos> gmaxwell: i'd be strongly in favor of a separate min mining feerate 05:10 < morcos> do you actually want to remove blockmaxsize (and blockmaxweight?) as options? 05:11 -!- MykelSIlver [~MykelSIlv@192-228-145-85.ftth.glasoperator.nl] has joined #bitcoin-core-dev 05:11 < morcos> i don't feel strongly about that, but i do think we should avoid setting blockmaxsize as default. i tried to benchmark the behavior and didn't show it being a big performance hit 05:11 < morcos> but thats counterintuitive 05:12 < morcos> and i'd rather not risk it by default 05:12 < gmaxwell> I think we should default them to maximum at a minimum. Removing them-- I don't really care probably removing them would irritate someone, so not worth doing. I don't think they're useful controls (beyond overriding our dumb maximum) 05:12 < morcos> right.. so default no setting for blockmaxsize in particular (to avoid size accounting unless you set it) 05:12 < gmaxwell> right. 05:13 < morcos> wumpus: you can backport just the first commit of #9239. i will separately test, but that was the intent. the only difference will be you will slide the slider to 1 but it will give you an answer for 2 05:13 < gribble> https://github.com/bitcoin/bitcoin/issues/9239 | Disable fee estimates for 1 block target by morcos · Pull Request #9239 · bitcoin/bitcoin · GitHub 05:14 < morcos> which is exactly what happens most of the time now anyway.. so will look like no change in behavior 05:14 < gmaxwell> Today, with BIP152 and fibre propagation has very little to do with blockmaxsize. And blockmaxweight is the wrong dimension for applying backpressure on fees: it's blind to demand, you'll still dumbly produce smaller blocks when there is a nice backlog and dumly produce blocks that are too big when there is none (at least the minfee stops the bleeding there) 05:15 * luke-jr notes that >1 MB blocks are still a problem as-is; it's not like gradual mining min-fee is implemented yet 05:16 < gmaxwell> luke-jr: yes, but a pointless softcap that virtually everone overrides is still not helping. 05:16 < luke-jr> gmaxwell: nobody overrides it >1 MB yet AFAIK. 05:17 < morcos> I think I went through this before, but can't see where I wrote it up. I think we actually need 3 minimum rates and minrelaytxfee (a default minimum for the mempool) is not one of them 05:17 < morcos> 1) min mining feerate 05:17 < morcos> 2) rate used to define dust (should change rarely) 05:17 < morcos> 3) rate used as the minimum increment to pay for relay bandwidth (closest analog to existing minrelaytxfee) 05:18 < luke-jr> morcos: how would 3 be different from current minrelaytxfee? 05:18 < morcos> I don't think 3) actually needs to a have a floor other than 0, but i don't suppose it hurts 05:18 < gmaxwell> luke-jr: they all will as soon as it matters, we trained them to with an unreasonable default. Last non-empty block that was under 990k was 217 blocks ago. 05:19 < luke-jr> as soon as it matters, would still be better than immediately by default 05:20 < gmaxwell> morcos: having a floor makes the system behavior more stable in any case, no real reason to go forwarding around low fee stuff that won't get mined ever just because we have nothing better to relay. :) 05:20 < morcos> although i guess, having a user set minimum for your mempool might be something people want (and a good fail safe if we get something wrong) so maybe we do want all 4 05:20 -!- aalex [~aalex@64.187.177.58] has joined #bitcoin-core-dev 05:21 < morcos> gmaxwell: sure, but its inherently rate limited by the mempool min fee.. but i agree. i don't know what i'm arguing that point... and separating 3 and 4 is the least important concern. 05:21 < gmaxwell> luke-jr: yes it is worse because it puts everyone in the mode of just nailing it to the maximum, which would undermine doing something sensible and dynamic. 05:22 < luke-jr> but we don't have anything sensible and dynamic yet. 05:22 < morcos> i guess the only reason it matters is if someone wants to set their minimum higher to say 10 sat/byte, then thats not clear that they really want their increment requirement to be 10 sat/byte 05:24 < gmaxwell> luke-jr: of course not, we have a nearly useless setting instead which you spend a lot of effort defending. This impedes doing something smarter. 05:24 < luke-jr> it doesn't impede anything.. 05:25 < gmaxwell> I promise it does, touching any of this means arguing with you and few people have interest in that. 05:25 < gmaxwell> and there is the polite lie that something already is there, there is a blocksize setting, you can make it larger or smaller, problem solved. 05:26 < jonasschnelli> I think the HD chain splitting will not be backward compatible. Adding a flag to CKeyPool would result in possible external keys from the internal chain on older versions of core 05:27 < jonasschnelli> Also, CKeyPool has no version-int. 05:27 < luke-jr> [12:22:27] jonasschnelli: is it just me, or is 0a261b63fd4f1b07431f8a65762ef9f1ef1c11c8 a bugfix that should be backported? 05:27 -!- aalex [~aalex@64.187.177.58] has quit [Ping timeout: 268 seconds] 05:27 < gmaxwell> I said it would be incompatible a bunch of times. 05:28 < jonasschnelli> I though we could make it compatible.. but yes. I guess it wont. 05:29 < jonasschnelli> Then I try to add a new record type, maybe "hdkey" that stores the keypath (int / ext chain) as well as the masterkey and the according pubkey. 05:29 < jonasschnelli> GetKey could derive the priv key on the fly 05:30 < jonasschnelli> luke-jr: we could bp... but o 05:30 < jonasschnelli> but i guess its not an important fix 05:31 -!- laurentmt [~Thunderbi@80.215.234.136] has joined #bitcoin-core-dev 05:32 < gmaxwell> what I was commenting on before was that we probably want to do several incompatible changes at once, so we don't end up having to support dozens of old versions. 05:33 < jonasschnelli> gmaxwell: Yes. That would be preferable. I think the splitting should be combined with the on-thy-fly derivation and the flexible keypath. 05:33 < jonasschnelli> Ideally +pub-CKD 05:34 < jcorgan> i haven't kept up recently, but are there any relevant BIPS or defacto practices from other wallets we should be paying attention to in this area? 05:35 < jonasschnelli> bip44/45 maybe. But I don't think its wise to force users to do pub key derivation. 05:36 < jonasschnelli> a flexible would allow to use bip44 and we could still stick to m/0'/k' by default 05:36 < jonasschnelli> a flexible keypath 05:37 < jcorgan> reviewing the flexpath PR is on my list this morning 05:38 < jonasschnelli> thanks 05:39 < jcorgan> but, i'm mostly just wondering if there are any good practices from other wallets we might benefit from understanding 05:39 < jcorgan> just thinking out loud 05:40 -!- laurentmt [~Thunderbi@80.215.234.136] has quit [Quit: laurentmt] 05:43 < gmaxwell> Doing public derrivation while the software supports private key export is really extremely inadvisable. 05:43 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has joined #bitcoin-core-dev 05:44 < gmaxwell> I also don't think it makes sense to spend time on advanced features when basic functionality is still kinda broken. 05:44 < gmaxwell> but I'm not the one working on it. :) 05:44 < jcorgan> agree, but it's important to define what unbroken basic functionality should be :) 05:45 < jcorgan> in other words, if there is to be breaking changes, and you only want to break it once, better get it right 05:46 < jcorgan> i didn't mean to jump into the middle of your thread, just noticed it and taking an interest. carry on. 05:47 < jonasschnelli> I agree with gmaxwell. We first need to solve the key-chain split. 05:48 < jonasschnelli> child key derivation is an interesting feature if you want to use core together with a hardware wallet. 05:48 < jonasschnelli> child pub key d. 05:50 < bitcoin-git> [bitcoin] luke-jr opened pull request #9266: Qt/RPCConsole: Put column enum in the right places (master...bugfix_datarole) https://github.com/bitcoin/bitcoin/pull/9266 05:54 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has joined #bitcoin-core-dev 06:02 -!- paveljanik [~paveljani@79.98.72.176] has joined #bitcoin-core-dev 06:02 -!- paveljanik [~paveljani@79.98.72.176] has quit [Changing host] 06:02 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev 06:29 -!- aalex [~aalex@64.187.177.58] has joined #bitcoin-core-dev 06:34 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has joined #bitcoin-core-dev 06:43 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 260 seconds] 06:49 -!- fanquake [~fanquake@unaffiliated/fanquake] has quit [Quit: Leaving.] 06:50 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 06:50 -!- wasi [~wasi@gateway/tor-sasl/wasi] has quit [Remote host closed the connection] 06:51 -!- wasi [~wasi@gateway/tor-sasl/wasi] has joined #bitcoin-core-dev 06:56 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/3fbf07926240...31bcc667863f 06:56 < bitcoin-git> bitcoin/master fe37fbe Wladimir J. van der Laan: bitcoin-cli: Make error message less confusing... 06:56 < bitcoin-git> bitcoin/master 31bcc66 MarcoFalke: Merge #9265: bitcoin-cli: Make error message less confusing... 06:56 < bitcoin-git> [bitcoin] MarcoFalke closed pull request #9265: bitcoin-cli: Make error message less confusing (master...2016_12_rpccli_message) https://github.com/bitcoin/bitcoin/pull/9265 06:59 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/31bcc667863f...5412c08c3cf1 06:59 < bitcoin-git> bitcoin/master b7aa290 S. Matthew English: unification of Bloom filter representation... 06:59 < bitcoin-git> bitcoin/master 5412c08 MarcoFalke: Merge #9223: unification of Bloom filter representation... 06:59 < bitcoin-git> [bitcoin] MarcoFalke closed pull request #9223: unification of Bloom filter representation (master...patch-10) https://github.com/bitcoin/bitcoin/pull/9223 07:21 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5412c08c3cf1...98514988a3d3 07:21 < bitcoin-git> bitcoin/master 08ed8c1 Gregory Maxwell: Developer docs about existing subtrees.... 07:21 < bitcoin-git> bitcoin/master 9851498 MarcoFalke: Merge #9246: Developer docs about existing subtrees.... 07:21 < bitcoin-git> [bitcoin] MarcoFalke closed pull request #9246: Developer docs about existing subtrees. (master...devdocs_for_subtrees) https://github.com/bitcoin/bitcoin/pull/9246 07:25 -!- bsm117532 [~mcelrath@38.121.165.30] has quit [Read error: Connection reset by peer] 07:26 -!- bsm117532 [~mcelrath@38.121.165.30] has joined #bitcoin-core-dev 07:41 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/98514988a3d3...d7ba4a233bd5 07:41 < bitcoin-git> bitcoin/master 0828619 Suhas Daftuar: [qa] Dump debug logs on travis failures. 07:41 < bitcoin-git> bitcoin/master d7ba4a2 MarcoFalke: Merge #9257: [qa] Dump debug logs on travis failures.... 07:42 < bitcoin-git> [bitcoin] MarcoFalke closed pull request #9257: [qa] Dump debug logs on travis failures. (master...travis-debug-logs) https://github.com/bitcoin/bitcoin/pull/9257 07:43 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev 07:44 -!- fengling [~fengling@223.223.187.142] has quit [Read error: Connection reset by peer] 07:45 < bitcoin-git> [bitcoin] morcos opened pull request #9267: Disable fee estimates for a confirm target of 1 block (0.13...backport9239) https://github.com/bitcoin/bitcoin/pull/9267 07:45 -!- fengling [~fengling@223.223.187.142] has joined #bitcoin-core-dev 07:45 -!- Magma [~magma@magma.tokyo] has quit [Read error: Connection reset by peer] 07:45 -!- Magma [~magma@magma.tokyo] has joined #bitcoin-core-dev 07:53 -!- Magma [~magma@magma.tokyo] has quit [Read error: Connection reset by peer] 07:55 -!- abpa [~abpa@2602:306:b837:dbf0:10d2:337:7ea0:6306] has joined #bitcoin-core-dev 07:55 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 07:56 -!- abpa [~abpa@2602:306:b837:dbf0:10d2:337:7ea0:6306] has quit [Client Quit] 07:58 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 268 seconds] 08:06 -!- laurentmt [~Thunderbi@80.215.234.136] has joined #bitcoin-core-dev 08:06 -!- laurentmt [~Thunderbi@80.215.234.136] has quit [Client Quit] 08:08 -!- Netsplit *.net <-> *.split quits: fengling 08:10 < MarcoFalke> sipa: I think this is needed to get our subtree clean again 08:10 < MarcoFalke> https://github.com/bitcoin-core/ctaes/pull/5 08:11 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has quit [Ping timeout: 244 seconds] 08:24 -!- laurentmt [~Thunderbi@80.215.234.136] has joined #bitcoin-core-dev 08:25 -!- laurentmt [~Thunderbi@80.215.234.136] has quit [Client Quit] 08:26 < instagibbs> can anyone explain why this isn't setting the argument correctly?: 08:26 < instagibbs> self.extra_args = [['-usehd={:d}'.format(i%2==0)] for i in range(4)] 08:26 < instagibbs> + self.extra_args[0].append("-limitancestorcount=10") 08:26 < instagibbs> I'm trying to set a custom limit in test, and it's not being enforced 08:30 -!- 44UAAAJXO [~magma@magma.tokyo] has joined #bitcoin-core-dev 08:30 -!- fengling [~fengling@223.223.187.142] has joined #bitcoin-core-dev 08:31 < instagibbs> ahhhh nevermind, the node is being spun down and up later in the test 08:32 -!- laurentmt [~Thunderbi@80.215.234.136] has joined #bitcoin-core-dev 08:35 -!- laurentmt [~Thunderbi@80.215.234.136] has quit [Client Quit] 08:37 < MarcoFalke> Yeah, we should use self.extra_args as argument where possible, instead of writing out the args every time. 08:43 -!- Magma [~magma@magma.tokyo] has joined #bitcoin-core-dev 08:44 -!- 44UAAAJXO [~magma@magma.tokyo] has quit [Ping timeout: 257 seconds] 08:45 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has joined #bitcoin-core-dev 08:47 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Quit: bye] 08:49 -!- jtimon [~quassel@186.31.134.37.dynamic.jazztel.es] has quit [Ping timeout: 260 seconds] 08:58 -!- BashCo_ [~BashCo@unaffiliated/bashco] has quit [Remote host closed the connection] 08:59 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 268 seconds] 09:03 < instagibbs> sdaftuar, how does one get the cached value? 09:03 < sdaftuar> instagibbs: oy, i'm thinking about it now. ancestor count is easy, but the descenadnt count is not... 09:03 < instagibbs> how often will that one trigger in wallet context though? 09:03 < sdaftuar> unfortunately the test we want to do is, what is the max number of descendants that any of pcoin's ancestors have? 09:04 < sdaftuar> and also, i think we should compare those values to some lower thresholds (say 10 instead of 25) 09:04 < instagibbs> I'm finding bugs with my implementation with non-standard values 09:05 < sdaftuar> well i think your implementation would never skip over any pcoins? except if the chain limits are violated in a reorg 09:05 < instagibbs> it always only fails at the 25 threshhold, unsure why. If we have an ancestor one already cached, we might just use that 09:05 < instagibbs> seems less error-prone, and should catch the common case? 09:06 < instagibbs> im not sure what you mean? 09:06 < sdaftuar> so to answer your first question, each mempool entry has a value nCountWithAncestors 09:06 < instagibbs> how do I access that entry 09:06 < instagibbs> (looking) 09:07 < sdaftuar> instagibbs: might need to add some kind of accessor, but mempool.mapTx.find(txid) or something should do it. 09:08 < sdaftuar> since we're already checking to see that it's in the mempool, this seems okay to me 09:08 < sdaftuar> CalculateMemPoolAncestors() can be a bit expensive, fyi 09:09 < sdaftuar> but in order to do the descendant calculation properly, we'd need to call CMPA on the pcoins in question, and then check the descendant count of each ancestor returned 09:09 < sdaftuar> (which i guess is equivalent to calling CMPA with lower thresholds) 09:09 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 09:10 < sdaftuar> i'm not sure though how reasonable it is to call CMPA inside AvailableCoins? if somehow your wallet has a lot of in-mempool stuff, this could be slow. not sure how to think about it. 09:11 < instagibbs> I don't understand. I likely don't understand what CMPA is actually doing. I thought it was already checking for limit violations 09:12 < sdaftuar> you're trying to see if a transaction that spends pcoins would be a limit violation. you're checking to see if pcoins itself would fail to get into the mempool based on the configured limits. 09:12 < sdaftuar> but you know it's already in the mempool 09:12 < sdaftuar> so almost by definition, those limits won't be violated 09:12 < sdaftuar> (turns out there is some weird edge case behavior where it could be, but i think that's beside the point) 09:14 < sdaftuar> oh, maybe i didn't make this clear: we call CMPA when we try to accept a new tx to the mempool. so if it's already gotten in -- a precondition of your code -- then calling it again on that tx, with the same limits, really shouldn't fail. 09:18 < instagibbs> ok now I'm confused why this works in the general case then. 09:18 < instagibbs> with default values 09:18 < sdaftuar> hmm. do you have a test i can look at? 09:19 -!- laurentmt [~Thunderbi@80.215.234.136] has joined #bitcoin-core-dev 09:19 < instagibbs> yes but its very easy to test the base case 09:19 < instagibbs> let me push my test im working on 09:20 -!- laurentmt [~Thunderbi@80.215.234.136] has quit [Client Quit] 09:20 < instagibbs> just send all your coins to yourself in a loop 26 times 09:21 < instagibbs> last time will trigger ATMP failure without these lines, with it triggers "Insufficient funds" 09:21 < sdaftuar> interesting, i'll take a look 09:25 < instagibbs> you're right though, this makes no sense in retrospect 09:28 * sdaftuar just learned about set_trace, whoa! 09:28 < instagibbs> wait... how do you write all those tests o_0 09:28 < sdaftuar> slowly :) 09:29 < instagibbs> i think it spawns zombies if you run in batch mode, be careful :P 09:30 < instagibbs> that test has "10" as the limit, but it successfully sends 25 times 09:30 < instagibbs> no matter what it's set to 09:30 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 09:30 < instagibbs> and the error message changes to mempool entry failure if you set a different value... 09:31 < sdaftuar> ah i think i see what's happening. 09:32 < sdaftuar> there might be some edge case bugs in CMPA we ought to fix 09:33 < sdaftuar> oh but it looks like for limit purposes, CMPA is trying to calculate whether adding the tx given would cause the limits to be exceeded. 09:33 < sdaftuar> ugh, this is kind of messy 09:34 < gmaxwell> sdaftuar: you're clear on the goal right? don't consider any coin we couldn't actually spend. 09:34 < sdaftuar> gmaxwell: yes, understood. 09:35 < gmaxwell> I wouldn't worry much about the coinselection being slow if you have many unspent inputs-- so long as slow isn't so slow as to cause rpc timeouts. 09:36 < gmaxwell> The code that figured out if all the parents were confirmed or ismine used to have factorial complexity... nice natural rate limitor on building large unconfirmed chains. :) 09:36 < sdaftuar> gmaxwell: instagibbs: i have two approaches to suggest, not sure which is better: 09:37 < sdaftuar> 1) do basically what you do here, except using CMPA correctly (i'd need to figure out exactly how to do that, certainly you could pass in a tx that spends pcoins, or maybe there's a way to call CMPA with adjusted limit values that works, not sure) 09:38 < sdaftuar> 2) don't bother calling CMPA in AvailableCoins, and instead just check to see if pcoins has more than N ancestors (a cached value) for some N much less than the default ancestor limit (maybe 5 or 10). 09:38 < sdaftuar> and then find a later point in the wallet code to abort the send if the descendant count would be violated 09:39 < instagibbs> I like (2) for now, just because I can't tell how CMPA is working sometimes :) 09:39 < gmaxwell> for the case that people hit, really it's just the ancestor limit that actually matters: people chaining change. 09:39 < instagibbs> yep 09:39 < sdaftuar> yeah the problem with 1) is that you don't know the size of the tx you're generating, so it's always going to be possible for the tx to ultimately fail 09:40 < sdaftuar> 2) is easy, but may result in some rare annoyances 09:40 < gmaxwell> sdaftuar: this is a sign we should reconsider that relay policy... the size is cornercase enough that it shouldn't really matter. 09:41 < sdaftuar> gmaxwell: we knew the descendant count/size was a theoretical problem from the start (someone else spending outputs of a tx that pays to you can prevent you from relaying new transactions until those other ones clear), but i believe it's the 09:42 < sdaftuar> descendant count/size stuff that limits DoS attacks on the mempool the most 09:42 < instagibbs> Ok, I'm going to learn more about CMPA, then likely do (2) instead 09:42 < sdaftuar> "those other ones clear" <--- should be until the parent confirms 09:44 < gmaxwell> sdaftuar: why do you suggest above going 'much less' than the ancestor limit? 09:45 < sdaftuar> gmaxwell: i just think that we should steer well clear of the relay policy limits. partly just philosophy (we didn't think there were any common use cases close the values we chose) 09:46 -!- bitcoin837 [369924af@gateway/web/freenode/ip.54.153.36.175] has joined #bitcoin-core-dev 09:46 < sdaftuar> but also practically, being near the ancestor limit makes it more likely some peer will think you're violating, say, the descendant limit 09:46 < bitcoin837> hey guys, before I write my own, does anyone already have a bash script that interacts with bitcoin-cli to do a sendmany split of a large chunk to your own deposit addresses? 09:46 < gmaxwell> makes sense. well doing 5 less would probably provide plenty of safty there. 09:46 < sdaftuar> (breaking relay) 09:48 < instagibbs> bitcoin837, #bitcoin 09:48 < bitcoin837> sure 09:48 < instagibbs> np 09:53 -!- laurentmt [~Thunderbi@80.215.234.136] has joined #bitcoin-core-dev 10:07 < sdaftuar> instagibbs: gmaxwell: there's a problem with this approach 10:08 < sdaftuar> just talked to morcos, and one thing we realized is that you might be combining lots of inputs, each with many ancestors 10:08 < sdaftuar> so the resulting transaction would fail 10:08 * BlueMatt sooo doesnt want to have to rebase #9260...anyone wanna review? 10:08 < gribble> https://github.com/bitcoin/bitcoin/issues/9260 | Mrs Peacock in The Library with The Candlestick (killed main.{h,cpp}) by TheBlueMatt · Pull Request #9260 · bitcoin/bitcoin · GitHub 10:09 < sdaftuar> so the best thing we could do in AvailableCoins is eliminate any coins that are already at the ancestor limit, i guess. but we need additional logic in SelectCoins..() i think 10:09 -!- bitcoin837 [369924af@gateway/web/freenode/ip.54.153.36.175] has quit [Ping timeout: 260 seconds] 10:11 < instagibbs> wait, why is this different than what we were going to do? 10:11 < instagibbs> i thought that was the definition of ancestor number 10:11 < instagibbs> oh i see, right 10:12 < instagibbs> I had this thought a couple minutes ago and somehow lost it. Yes, 2 inputs may have n-1 history, making 2n-2 10:12 < sdaftuar> yep. or making n-1, if they're different outputs of the same tx 10:12 < sdaftuar> we have to check later on 10:12 < instagibbs> Yep. glad someone else thought of it and actually communicated 10:13 < gmaxwell> In the case people have actually encountered, it's just a decendant chain limit AFAIK. Many of the other cases would be pretty hard to hit with the wallet behavior. 10:14 < instagibbs> does the wallet prefer shorter unconfirmed chains vs longer? 10:14 < instagibbs> I know it prefers confirmed change first, etc 10:14 < gmaxwell> It is completely blind to it right now. 10:14 < instagibbs> ok, so it might not be all that rare 10:14 < gmaxwell> To make it prefer shorter is easy: first make it so it can reject if it's too long, and try again with a higher limit if that fails. 10:15 < sdaftuar> i think one way we could do that is to make SelectCoinsMinConf do the checking against some passed in limit, and then call it twice or something 10:15 < sdaftuar> we already call SelectCoinsMinConf repeatedly 10:15 < gmaxwell> thats how we handled unconfirmed, yep. 10:15 < instagibbs> yeah its the same idea 10:16 < instagibbs> prefer deep confirmed coins from others, prefer confirmed coins, ok now try unconfirmed <-- I think today 10:16 < sdaftuar> the other approach would be if there's a way to do it directly in ApproximateBestSubset? 10:16 < gmaxwell> it does 6,1,0 and it could become 6,1,0+1i,0+10i,0+20i. 10:16 < sdaftuar> not sure if that's a good idea 10:17 < instagibbs> ABS assumes you have enough coins i believe 10:17 < instagibbs> which means if you run into too long chain, that will be different and fail 10:18 < gmaxwell> not a great idea to do it in ABS. Just making it a parameter to SelectCoinsMinConf would be straightforward. 10:19 < instagibbs> gmaxwell, why are we using complex numbers for the arg :P 10:19 < sdaftuar> ok, so if we do it in SCMC, then i think the approach would be to filter out of availableCoins the ones with >N ancestors, and testing whether the resulting set of inputs would pass CMPA before returning? 10:19 < sdaftuar> does that sound right? 10:22 < sdaftuar> hmm. this doens't quite make sense -- once ABS returns coins with enough value, but that violate the chain limits, trying again isn't likely to help, is it? 10:23 < instagibbs> SCMF can say "no" 10:23 < instagibbs> pass failure to SC, returning Insufficent Funds 10:23 < sdaftuar> right, but what is going to do differnetly when invoked with different limits? 10:24 < gmaxwell> sdaftuar: you start with the most restrictive limits first. 10:24 < gmaxwell> and it will only consider coins which will not violate. 10:25 < instagibbs> if it fails to get enough coins under each limit, raise it, if it can never get enough, fails 10:27 < gmaxwell> and if that can't find a solution, you relax the limits and try again. The first limit is very confirmed, then confirmed, then unconfirmed but 1 deep, then unconfirmed and deeper... The reason to try with multiple limits is so that it doesn't do something dumb like build a chain of 19 deep, then at the 20's also spend all your other unconfirmed coins... so that the next call has nothing to spe 10:27 < gmaxwell> nd. 10:29 < sdaftuar> gmaxwell: i'm trying to understand how ABS works now, but it's not obvious to me how effective it would be at randomly finding a set of inputs that don't violate the chain limits as the set of available coins it's given increases 10:29 < instagibbs> ABS wouldnt be in charge of it, would have to be higher 10:29 < instagibbs> ABS I believe just has a set of inputs, and tries to make "good" fits based on value 10:30 < sdaftuar> instagibbs: right. so let's say you have a set of inputs with at most 1 ancestors that has enough value to createa a tx, but the resulting tx has too many ancestors 10:30 < sdaftuar> what are the chances that when you add some more inputs (where it is possible to find a suitable input set that passes the chain limits), that ABS will return it to you? 10:30 < instagibbs> hm the issue seems to continue all the way until you create the actual transaction 10:30 < gmaxwell> yes, there is nothing really we can do about the ancestor limit right now-- it even depends on future and non-local information, we can deal with the decendant limit now. 10:31 < sdaftuar> gmaxwell: did you swap ancestor and descendant in that last line? 10:31 < gmaxwell> yes. 10:32 < gmaxwell> Oh you're also saying for ancestors. Indeed, we can only be greedy, you're right. 10:32 < sdaftuar> if ABS were somehow aware of the constraint, we might get lucky and succeed 10:32 < sdaftuar> but if it's not... 10:33 < gmaxwell> it's not a framework that would likely do well with that kind of constraint in any case. 10:33 < gmaxwell> For some reason I thought the combination rule for ancestor limit was MAX (a depth) not sum. 10:34 < instagibbs> Yeah me too :/ 10:34 < sdaftuar> sum reflects how much recordkeeping (and pointer hopping) we do 10:34 < instagibbs> So, we could still have the wallet prefer shorter inputs, and post-transaction finalization, have a CMPA check before returning? 10:35 < instagibbs> shorter-chain inputs greedily* 10:35 < sdaftuar> yes i still think it'd be useful to do what i wrote above: in SCMC, filter out from AvailableCoins the ones that have more than N ancestors, for different values of N 10:35 < gmaxwell> sdaftuar: yes, while realizing that I was wrong from your comment it was clear enough to me why it's sum. 10:36 < sdaftuar> but there are then two failures from SCMC: not enough value, in which case it's worth tryign again with a higher limit; or chain limit exceeded, in which case I'm not sure it's worth calling again with a higher limit 10:37 < sdaftuar> we might fail sometimes to generate a transaction when there are coins available that would work, but this seems like the best we can do right now 10:37 < sdaftuar> and certainly the most important thing is to not commit a transaction that then fails in ATMP 10:38 < sdaftuar> "This must not fail." :) 10:40 < instagibbs> so we'll still need a CMPA call, or something similar, to get the actual ancestor count before returning right 10:40 < instagibbs> otherwise "this can fail" :) 10:40 < sdaftuar> yes. once we've put the transaction together, we can call CMPA and know whether it'll pass. 10:40 < sipa> i think we should deal correctly with a failed ATMP call, like removing the tx from the wallet in that case 10:41 < sipa> (in addition to doing sanity checks ahead of time) 10:41 < sdaftuar> sipa: agreed 10:41 < instagibbs> what's the barrier to that fix? I don't know the wallet well enough 10:42 < sdaftuar> i don't think we ever delete anything now, do we? 10:42 < instagibbs> removeprunedfunds :P 10:42 < instagibbs> but no, not normallly 10:43 < sdaftuar> oh, neat. i didn't know that existed! 10:43 < instagibbs> it's meant to be for importing/removing proofs of payment without scanning 10:44 < morcos> is the reason we commit first in case the node crashes? 10:44 < sdaftuar> so we can just zap tx's if we create them but ATMP fails? that seems easy 10:45 < morcos> we could alwasy have ATMP(fJustCheck) 10:45 < sdaftuar> morcos: i assumed that's why we commit first 10:45 -!- laurentmt [~Thunderbi@80.215.234.136] has quit [Ping timeout: 260 seconds] 10:46 < sdaftuar> morcos: ATMP(fJustCheck) doesn't work very well with RBF and mempool eviction 10:46 < instagibbs> morcos, hmm I thought there was issues with that, otherwise we'd have an easy way to check in rpc if policy is ok with it? 10:46 < instagibbs> ^ ok that 10:46 < morcos> ah, yes... 10:46 < sdaftuar> maybe RBF isn't a big deal, not sure 10:47 < gmaxwell> someone should just remove that comment, it's not true. "Must not fail perminantly" it's fine if it fails until some transactions confirm. 10:47 < morcos> ok.. well we just want to be a bit careful.. b/c even things that do get rejected from ATMP, first make it into the memory pool. so we'll want to make sure no one ever makes it so those coudl get relayed or remembered in some other way, if we're about nuke them from teh wallet 10:47 < gmaxwell> right, it's important to never remove something from the wallet that could have relayed. 10:48 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 258 seconds] 10:48 < morcos> or remembered in the future opportunistic eviction rejection map 10:48 < sdaftuar> morcos: oh! now i see what you mean. 10:49 < gmaxwell> ::sigh:: 10:50 < gmaxwell> The thing I was hoping this issue would fix is the wallet needlessly maining excessively deep transactions when it could choose inputs and avoid it. 10:51 < instagibbs> we can still do that, yes? 10:51 < gmaxwell> Removing things from the wallet on ATMP failure might be fine (though if it'll relay after the next block ... it might have been better to just leave it), but it won't avoid needlessly creating unattractive transactions. 10:51 < gmaxwell> instagibbs: sure. 10:53 < morcos> gmaxwell: yeah i think we get it, we're jsut trying to fix up the edges too 10:53 -!- MykelSIlver [~MykelSIlv@192-228-145-85.ftth.glasoperator.nl] has quit [Remote host closed the connection] 10:53 < morcos> but as i think you mentioned the other day, as it is now it won't get relayed after the next block, b/c you don't rebroadcast whats not in your mempool and you don't try to reaccept whats in your wallet 10:54 < instagibbs> is this referring to ATMP failed transactions? 10:54 < morcos> but we all agree that first you try with things that in the simple case (single stranded chains) will stay well clear of the limits 10:55 < sdaftuar> instagibbs: yeah there's a bug now, where the code that tries to periodically relay unconfirmed wallet transactions skips over things that aren't in the mempool 10:55 < sdaftuar> instagibbs: i think it's an easy fix, just try to reaccept to the mempool first 10:56 < morcos> sdaftuar: i dont know if i 100% agree thats a bug, but certainly a separate issue 10:56 < gmaxwell> We we should fix rebroadcast transactions to try remempolling things. :) 10:57 < sdaftuar> morcos: right, i guess reaccepting makes it harder for you to abandon a too-low-fee tx? 10:57 < gmaxwell> thats a day one bug, considering reorgs/doublespends. 10:57 -!- MykelSIlver [~MykelSIlv@192-228-145-85.ftth.glasoperator.nl] has joined #bitcoin-core-dev 10:57 < morcos> gmaxwell: at the risk of tangents.. the danger with that is that i think we'll go into these endless work loops of trying things that have long since been double spent or whatever 10:58 < morcos> sdaftuar: no its easy enough to skip abandoned, we already skip reaccepting those on startup 10:58 < morcos> but yes, it makes the auto-defacto-abandoned state disappear, which is not necessarily a good thing 10:58 < instagibbs> ok, I need coffee bbl 10:58 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 10:59 < gmaxwell> morcos: well it's once per half an hour or whatever... and work per transaction. We also can tell when transactions are conflicted and could skip those. 11:02 < morcos> gmaxwell: i guess we could just ask some users with big wallets to see how long that takes them on startup (maybe we need to put in benching for it), since we already do it then 11:03 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has joined #bitcoin-core-dev 11:07 -!- MykelSIlver [~MykelSIlv@192-228-145-85.ftth.glasoperator.nl] has quit [Quit: -a- Android IRC 2.1.17] 11:07 -!- MykelSIlver [~MykelSIlv@192-228-145-85.ftth.glasoperator.nl] has joined #bitcoin-core-dev 11:08 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 260 seconds] 11:08 -!- Chris_Stewart_5 [~Chris_Ste@gateway/vpn/privateinternetaccess/chrisstewart5/x-62865615] has joined #bitcoin-core-dev 11:10 < gmaxwell> morcos: the non-mempool part of resend wallet txn wouldn't be needed if we had mempool sync. :P 11:10 < morcos> if we get it right, we can just skip blocks and PoW too. :P :P 11:11 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 268 seconds] 11:17 -!- jtimon [~quassel@186.31.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev 11:17 -!- laurentmt [~Thunderbi@80.215.234.136] has joined #bitcoin-core-dev 11:18 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/d7ba4a233bd5...9e4bb312e695 11:18 < bitcoin-git> bitcoin/master facbfa5 MarcoFalke: [qa] Get rid of duplicate code 11:18 < bitcoin-git> bitcoin/master 9e4bb31 MarcoFalke: Merge #9221: [qa] Get rid of duplicate code... 11:18 < bitcoin-git> [bitcoin] MarcoFalke closed pull request #9221: [qa] Get rid of duplicate code (master...Mf1611-qaMaxUploadDuplCode) https://github.com/bitcoin/bitcoin/pull/9221 11:18 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 11:19 -!- LeMiner [~LeMiner@unaffiliated/leminer] has quit [Read error: Connection reset by peer] 11:19 -!- LeMiner [~LeMiner@unaffiliated/leminer] has joined #bitcoin-core-dev 11:22 -!- laurentmt [~Thunderbi@80.215.234.136] has quit [Client Quit] 11:34 -!- pindarhk [sid105966@gateway/web/irccloud.com/x-rqnkxnaaggbdpofn] has quit [Remote host closed the connection] 11:34 -!- aspect_ [sid151486@gateway/web/irccloud.com/x-ehhckfzudnfhtowv] has quit [Remote host closed the connection] 11:34 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-dpoacivzzknmmqke] has quit [Remote host closed the connection] 11:34 -!- mappum [sid43795@gateway/web/irccloud.com/x-pvbatempfqkljggs] has quit [Remote host closed the connection] 11:34 -!- eragmus [sid136308@gateway/web/irccloud.com/x-ryomcompfzoohwlo] has quit [Remote host closed the connection] 11:34 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/9e4bb312e695...c36229b0b2e9 11:34 < bitcoin-git> bitcoin/master 8a70a9d wodry: Improvement of documentation of command line parameter 'whitelist' 11:34 < bitcoin-git> bitcoin/master c36229b MarcoFalke: Merge #9251: Improvement of documentation of command line parameter 'whitelist'... 11:35 < bitcoin-git> [bitcoin] MarcoFalke closed pull request #9251: Improvement of documentation of command line parameter 'whitelist' (master...patch-3) https://github.com/bitcoin/bitcoin/pull/9251 11:54 -!- mappum [sid43795@gateway/web/irccloud.com/x-qdonrgftqslebfgv] has joined #bitcoin-core-dev 11:59 < bitcoin-git> [bitcoin] MarcoFalke closed pull request #9064: unify capitalization of "bitcoin address" (master...changeCaps) https://github.com/bitcoin/bitcoin/pull/9064 12:08 -!- pindarhk [sid105966@gateway/web/irccloud.com/x-eubqzdmhxpqvwuzh] has joined #bitcoin-core-dev 12:10 -!- mrkent [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 12:11 -!- eragmus [sid136308@gateway/web/irccloud.com/x-pjsedpnxwmwrsfde] has joined #bitcoin-core-dev 12:15 -!- aspect_ [sid151486@gateway/web/irccloud.com/x-jkkhchzezvjcdbld] has joined #bitcoin-core-dev 12:19 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-svypifmqlfoimwwi] has joined #bitcoin-core-dev 12:22 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds] 12:25 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)] 12:28 < sdaftuar> BlueMatt: i was trying to verify that the second commit in #9260 is move only. first, do you have any clever ways to suggest that i could use to do that? 12:28 < gribble> https://github.com/bitcoin/bitcoin/issues/9260 | Mrs Peacock in The Library with The Candlestick (killed main.{h,cpp}) by TheBlueMatt · Pull Request #9260 · bitcoin/bitcoin · GitHub 12:28 < sdaftuar> second -- i tried to use git blame -C on the net_processing.cpp file, to verify that you didn't introduce any changes 12:29 < sdaftuar> that almost works, though for some reason i can't figure out, it assigns blame to you for a handful of lines around the Misbehaving() function 12:29 < sdaftuar> i can't see any change in that code, but i'm puzzled as to why git thinks a change was introduced 12:30 < sipa> sdaftuar: git diff --patience COMMIT~:oldfile COMMIT:newfile 12:31 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has joined #bitcoin-core-dev 12:31 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 12:33 < sdaftuar> sipa: thanks, i'll try that 12:35 -!- afk11 [~afk11@unaffiliated/afk11] has quit [Ping timeout: 265 seconds] 12:37 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 246 seconds] 12:40 < morcos> sdaftuar: its not ENTIRELY move only 12:40 < sdaftuar> yeah i know... just trying to figure out what the real diff is 12:41 < sipa> the only diff i saw was some header changes, and the snippet i pasted in a comment 12:41 -!- afk11 [~afk11@176.61.67.182] has joined #bitcoin-core-dev 12:41 -!- afk11 [~afk11@176.61.67.182] has quit [Changing host] 12:41 -!- afk11 [~afk11@unaffiliated/afk11] has joined #bitcoin-core-dev 12:43 < sdaftuar> sipa: i think there's a one-line change about feeFilterRounder too? 12:44 < sipa> sdaftuar: ah, he squashed in some changes 12:44 < sipa> i reviewed an earlier version 12:44 < sdaftuar> ah ok 12:44 < sipa> 84922e4 12:46 -!- Atomicat [~Atomicat@unaffiliated/atomicat] has quit [Ping timeout: 258 seconds] 12:48 < morcos> btw.. that "almost bug" with saferModifyNewCoins, i forgot i had a node running with my new coinsviewcache patches. it lost consensus a couple days ago.. 12:48 < morcos> investigating now, but i bet it hit that bug... whew. dodged a bullet 12:48 < sdaftuar> nice. lets not PR that one. :) 12:51 -!- Atomicat [~Atomicat@unaffiliated/atomicat] has joined #bitcoin-core-dev 12:55 -!- owowo [ovovo@gateway/vpn/mullvad/x-fbexxojwcqohamaj] has quit [Ping timeout: 244 seconds] 12:55 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has quit [Ping timeout: 256 seconds] 12:56 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 13:00 -!- aalex_ [~aalex@64.187.177.58] has joined #bitcoin-core-dev 13:03 < instagibbs> sdaftuar, ok updated my PR, thanks for the help 13:03 -!- aalex [~aalex@64.187.177.58] has quit [Ping timeout: 245 seconds] 13:05 < jonasschnelli> Isn't this a bug: https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/keypool.py#L40 13:05 < sdaftuar> instagibbs: cool, i'll take a look 13:05 < jonasschnelli> We fill the keypool with a new target size of 3 13:05 < jonasschnelli> then we manage to reserve 4 keys: https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/keypool.py#L45 13:08 -!- k0ng [~k0ngo@95.215.44.99] has quit [Quit: Leaving] 13:11 -!- owowo [ovovo@gateway/vpn/mullvad/x-umdvslmxdksiigoj] has joined #bitcoin-core-dev 13:13 < cfields> jonasschnelli: seems to always reserve target + 1 13:14 < cfields> ./bitcoin-cli -testnet keypoolrefill 104 13:14 < cfields> 2016-12-02 21:13:24 keypool added key 108, size=105 13:19 < sipa> i think that may be the vchDefaultKey... 13:25 -!- MykelSIlver [~MykelSIlv@192-228-145-85.ftth.glasoperator.nl] has quit [Quit: -a- Android IRC 2.1.17] 13:52 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Read error: Connection reset by peer] 13:53 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev 13:59 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds] 14:02 -!- Chris_Stewart_5 [~Chris_Ste@gateway/vpn/privateinternetaccess/chrisstewart5/x-62865615] has quit [Ping timeout: 250 seconds] 14:02 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 14:02 < MarcoFalke> no, don't think this is vchDefaultKey 14:03 < MarcoFalke> the target+1 comes from a time when the function was only used by one caller (Maybe getnewaddress or something, where you first call keypoolrefill and then grab one key) 14:04 < MarcoFalke> Now if you call it from another place which does not grab the key, you end up with off-by-one 14:08 -!- cheese_ [~x@c-174-54-219-36.hsd1.pa.comcast.net] has joined #bitcoin-core-dev 14:08 -!- cheese_ [~x@c-174-54-219-36.hsd1.pa.comcast.net] has quit [Changing host] 14:08 -!- cheese_ [~x@unaffiliated/cheeseo] has joined #bitcoin-core-dev 14:12 -!- RoyceX [~x@unaffiliated/cheeseo] has quit [Ping timeout: 250 seconds] 14:18 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 14:34 < instagibbs> sdaftuar: hmm the in mempool check done in AvailableCoins doesn't cover that case? 14:34 < instagibbs> Oh I see confirmation plus not mempool 14:38 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 244 seconds] 14:38 -!- aalex__ [~aalex@64.187.177.58] has joined #bitcoin-core-dev 14:42 -!- aalex_ [~aalex@64.187.177.58] has quit [Ping timeout: 248 seconds] 14:48 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has left #bitcoin-core-dev [] 14:50 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev 14:55 -!- owowo [ovovo@gateway/vpn/mullvad/x-umdvslmxdksiigoj] has quit [Ping timeout: 244 seconds] 15:00 -!- owowo [ovovo@gateway/vpn/mullvad/x-bskagjsotednaztd] has joined #bitcoin-core-dev 15:05 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev 15:11 -!- JackH [~laptop@79-73-189-171.dynamic.dsl.as9105.com] has quit [Remote host closed the connection] 15:20 < BlueMatt> wumpus: re: compilation memusage, validation.cpp takes ~1.19G, init ~1.1G, net_processing ~0.95G, rpcrawtx ~0.95G, rpcdump ~0.99G, rpcwallet ~1.0G, wallet ~1.2G, walletdb ~0.82G...main took ~1.46G 15:21 < BlueMatt> so while splitting main didnt cut out worst-case memusage by a /ton/, it did potentially make it possible to compile with 1.5G again, whereas it previously wasnt once you include the host 15:21 < sipa> but validation + net_processing together now take 2.14G! 15:21 < sipa> bad job! 15:21 < BlueMatt> lol 15:22 < luke-jr> need to do -flto with under 1 GB RAM use ‼‼‼‼! :p 15:22 < BlueMatt> luke-jr: I cant get lto to build anymore 15:25 -!- bsm117532 [~mcelrath@38.121.165.30] has quit [Remote host closed the connection] 15:26 < gmaxwell> BlueMatt: build time is greater for me too. 15:26 < BlueMatt> greater like slower or faster? 15:26 < luke-jr> it would be nice if you could compile "split LTO data" for libraries, so one can just use non-LTO for their system but have the ability to do LTO on demand 15:27 < gmaxwell> BlueMatt: slower. takes more time. 15:27 < BlueMatt> gmaxwell: yea, not surprising given how much shit we have in headers (and how much boost has in headers!) 15:27 < gmaxwell> bad except when napping is required. presumably its faster on things other than my laptop. 15:27 < gmaxwell> yea, need to get rid of boost. 15:27 < sipa> gmaxwell: are you using parallel lto linking? 15:28 < gmaxwell> sipa: no just the seperate files require more time in total. 15:28 < gmaxwell> due to all the garbage in headers. 15:28 < BlueMatt> gmaxwell: well even if it is slower, at least you dont have to wait for net_processing if you only want to touch validation or vica versa :p 15:28 < gmaxwell> I'm not complaining, its the right thing to do. 15:29 < gmaxwell> though the recompile benefit is mixed, it isn't all that often that I don't end up editing a header that causes everything to get recompiled. P 15:29 < gmaxwell> :P 15:29 < BlueMatt> i know, but I am complaining about headers in bitcoin core (and boost) 15:29 < sipa> gmaxwell: which gcc version? earlier gccs compiled both to object code and gimple in lto mode; in later ones the default is generating only gimple, which is faster 15:29 < BlueMatt> sipa: I dont think we're talking about lto 15:30 < sipa> ah. 15:36 -!- bsm117532 [~mcelrath@38.121.165.30] has joined #bitcoin-core-dev 15:37 < BlueMatt> sdaftuar: ok with not fixing your comments in #9260? 15:37 < gribble> https://github.com/bitcoin/bitcoin/issues/9260 | Mrs Peacock in The Library with The Candlestick (killed main.{h,cpp}) by TheBlueMatt · Pull Request #9260 · bitcoin/bitcoin · GitHub 15:37 < gmaxwell> I'm not talking about LTO. 15:39 -!- bsm117532 [~mcelrath@38.121.165.30] has quit [Client Quit] 15:41 < BlueMatt> paveljanik: you got a minute to ack 9260, since you already looked over it? 15:41 * BlueMatt is still hoping it can be merged prior to any other main.cpp changes :p 15:50 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has quit [Ping timeout: 268 seconds] 15:50 -!- justan0theruser [~justanoth@unaffiliated/justanotheruser] has joined #bitcoin-core-dev 15:58 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has quit [Quit: Textual IRC Client: www.textualapp.com] 16:01 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has joined #bitcoin-core-dev 16:04 -!- afk11 [~afk11@unaffiliated/afk11] has quit [Read error: Connection reset by peer] 16:05 -!- afk11 [~afk11@176.61.67.182] has joined #bitcoin-core-dev 16:05 -!- afk11 [~afk11@176.61.67.182] has quit [Changing host] 16:05 -!- afk11 [~afk11@unaffiliated/afk11] has joined #bitcoin-core-dev 16:30 -!- RoyceX [~x@unaffiliated/cheeseo] has joined #bitcoin-core-dev 16:31 -!- RoyceX [~x@unaffiliated/cheeseo] has quit [Read error: Connection reset by peer] 16:49 -!- mrkent [~textual@unaffiliated/mrkent] has quit [] 16:51 -!- mrkent [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 16:52 -!- mrkent [~textual@unaffiliated/mrkent] has quit [Client Quit] 16:52 -!- arowser [~quassel@106.120.101.38] has quit [Ping timeout: 256 seconds] 16:52 -!- mrkent [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 16:53 -!- mrkent [~textual@unaffiliated/mrkent] has quit [Client Quit] 16:54 -!- mrkent [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 16:54 -!- mrkent [~textual@unaffiliated/mrkent] has quit [Client Quit] 17:18 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection] 17:19 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev 17:21 -!- mrkent [~textual@unaffiliated/mrkent] has joined #bitcoin-core-dev 17:33 -!- mrkent [~textual@unaffiliated/mrkent] has quit [] 17:52 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-iwdwktcinbcydgue] has quit [Quit: Connection closed for inactivity] 17:53 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 18:03 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has joined #bitcoin-core-dev 18:18 -!- alpalp [~allen@cpe-24-27-58-209.austin.res.rr.com] has joined #bitcoin-core-dev 18:18 -!- alpalp [~allen@cpe-24-27-58-209.austin.res.rr.com] has quit [Changing host] 18:18 -!- alpalp [~allen@unaffiliated/alpalp] has joined #bitcoin-core-dev 18:25 < bitcoin-git> [bitcoin] sipa pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/c36229b0b2e9...2efcfa5acfac 18:25 < bitcoin-git> bitcoin/master 87c35f5 Matt Corallo: Remove orphan state wipe from UnloadBlockIndex.... 18:25 < bitcoin-git> bitcoin/master e736772 Matt Corallo: Move network-msg-processing code out of main to its own file 18:25 < bitcoin-git> bitcoin/master 76faa3c Matt Corallo: Rename the remaining main.{h,cpp} to validation.{h,cpp} 18:25 < bitcoin-git> [bitcoin] sipa closed pull request #9260: Mrs Peacock in The Library with The Candlestick (killed main.{h,cpp}) (master...net_processing_file) https://github.com/bitcoin/bitcoin/pull/9260 18:25 < cfields> let the rebasing begin :) 18:25 < sipa> now: the big rebasing game 18:26 < sipa> jinx 18:26 < cfields> haha 18:26 < BlueMatt> hey-o! 18:27 < cfields> BlueMatt: there are a few includes cleanups that can be done that should help with memory 18:27 < cfields> BlueMatt: not sure if you had a few in the queue, or if i should go ahead and PR 18:28 < BlueMatt> cfields: I dont have a queue of them, but was playing around at https://github.com/TheBlueMatt/bitcoin/commits/2016-12-memusage 18:28 < BlueMatt> feel free to peruse that and PR whatever you did 18:28 < sipa> damn, validation.cpp is *still* 8000 lines 18:28 < cfields> ok 18:29 < cfields> BlueMatt: boost/filesystem/path.hpp in validation.h was the only one i really had my eye on 18:30 < BlueMatt> now that I read the feedback, cfields' comments pointed out that filterRounder changed behavior just slightly - instead of rounding based on DEFAULT_MIN_RELAY_TX_FEE it now rounds based on -minrelaytxfee, oops 18:30 < BlueMatt> doesnt matter all that much, but the original behavior is probably more correct 18:30 < BlueMatt> welll, actually 18:31 < BlueMatt> hum, dunno, needs pointed out for discussion, not sure if its better that users can change that rounding or not 18:31 < BlueMatt> (given hardcoded-value-avoidance-policies) 18:31 < BlueMatt> I'll file an issue and open it up 18:31 < sipa> users changing the rounding is probably a slight privacy leak 18:31 < sipa> which is afaik the reason for rounding in the first place 18:31 < cfields> BlueMatt: hmm, i thought i verified the behavior didn't change there. Not sure how i noticed the mempool weirdness without seeing that too 18:31 < BlueMatt> it is, technically, but I'd be surprised if it wasnt already visible what your -minrelaytxfee is 18:32 < BlueMatt> but, ok, will pr a fix and see what folks say 18:36 < bitcoin-git> [bitcoin] TheBlueMatt opened pull request #9268: Fix rounding privacy leak introduced in #9260 (master...2016-12-feefilterrounder) https://github.com/bitcoin/bitcoin/pull/9268 18:39 * BlueMatt -> dinner, in a whole new world :p 18:44 < morcos> woo hoo! 18:45 * sipa gives BlueMatt a disney movie 18:45 -!- abpa [~abpa@2602:306:b837:dbf0:a93f:40a2:fe32:d4ea] has joined #bitcoin-core-dev 18:52 -!- fanquake [~fanquake@unaffiliated/fanquake] has joined #bitcoin-core-dev 18:52 -!- e4xit [~e4xit@cpc1-cmbg20-2-0-cust188.5-4.cable.virginm.net] has quit [Ping timeout: 240 seconds] 18:56 * jtimon still needs to google more about mrs peacock, but prefers to keep rebasing for now (I had things to rebase before renaming main, but one more step towards using git blame in main/validation) 19:00 < sipa> jtimon: do you know the game cluedo? 19:00 < jtimon> I don't think have played, but yes, it's famous 19:01 < sipa> the game ends when a player figures out who the murdered is, where the murder happened, and with what weapon 19:01 < jtimon> just enjoy with anything make main smaller 19:01 < sipa> *murderer 19:01 < alpalp> the game also ends if you guess wrong 19:01 < sipa> jtimon: well, main.cpp now has size 0 :p 19:01 < jtimon> right, calling main validation is going to take some time... 19:04 < jtimon> btw, I was serious about rewritting any part of https://github.com/bitcoin/bitcoin/pull/8328 that can be agreed on, just without any clear agreement, there's no point and keeping that open for long (since it's guaranteed to be arebase hell) 19:04 < jtimon> what about moving bitcoinconsensus.o from script to consensus? 19:05 < jtimon> anyway, first rebase 19:06 < sipa> BlueMatt: how do you measure compiler memory usage? 19:06 -!- e4xit [~e4xit@cpc1-cmbg20-2-0-cust188.5-4.cable.virginm.net] has joined #bitcoin-core-dev 19:16 -!- abpa [~abpa@2602:306:b837:dbf0:a93f:40a2:fe32:d4ea] has quit [Quit: Textual IRC Client: www.textualapp.com] 19:29 < cfields> BlueMatt: see what you think about https://github.com/theuni/bitcoin/commit/3f598dbe7100c7c6c7bfb7e10210585327ed9d31 19:29 < cfields> BlueMatt: I just hacked it up after the main split because it looked kinda trivial, not sure if that's a direction worth going in or not 19:29 < bitcoin-git> [bitcoin] sipa opened pull request #9269: Align struct COrphan definition (master...oneorphan) https://github.com/bitcoin/bitcoin/pull/9269 19:32 < sipa> BlueMatt: i succesfully compiled with flto, and i never saw memory usage of a single process go above ~700MB (i was just watching with top, this may not be very accurate) 19:32 < cfields> hmm, actually, CConnman could easily just hold the interface pointer, since it's abstract 19:32 < cfields> sipa: isn't lto much less intensive on the individual compilation units? 19:32 < cfields> or was that your point? 19:33 < sipa> yes 19:33 < cfields> ok, nm me 19:49 -!- wasi [~wasi@gateway/tor-sasl/wasi] has quit [Remote host closed the connection] 19:53 < sipa> BlueMatt: https://www.zerobin.net/?cd7cc1b7c1d2c689#/IYxSnkHTRRd5FdHJRRPwmRNZKdq+YPvKOW6rt0acTQ= 20:18 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 258 seconds] 20:24 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has quit [Ping timeout: 260 seconds] 20:25 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has joined #bitcoin-core-dev 20:26 < BlueMatt> sipa: you want grep VmPeak /proc/`pidof cc1plus`/status 20:30 < sipa> this is made using /usr/bin/time -f "%M" g++ 20:30 < BlueMatt> cfields: concept ack, but I'd say yes to just about anything that removed boost in any context 20:30 < BlueMatt> sipa: oh, even better 20:31 < sipa> BlueMatt: i made a tiny wrapper around g++ that calls time, and passed that as CXX= to configure 20:31 < BlueMatt> yea 20:33 < sipa> oh, the numbers for the binaries are likely inaccurate, as gcc spawns child processes that do the linking/compiling 20:34 < BlueMatt> yea, i mean thats where lto uses all its memory/time, though? 20:34 < sipa> time, yes 20:34 < sipa> memory, no 20:34 < BlueMatt> oh? the streaming shit? 20:35 < sipa> the memory used by those processes seems much lower than the typical first step compilation 20:36 -!- cryptapus is now known as cryptapus_afk 20:38 < sipa> like 60MB or so 20:39 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 20:50 -!- alpalp [~allen@unaffiliated/alpalp] has quit [Ping timeout: 240 seconds] 21:07 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #bitcoin-core-dev 21:11 -!- justan0theruser [~justanoth@unaffiliated/justanotheruser] has quit [Ping timeout: 256 seconds] 21:13 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has quit [Ping timeout: 258 seconds] 21:29 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection] 21:30 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev 21:47 < bitcoin-git> [bitcoin] jtimon opened pull request #9271: Discusion : 0.13 consensus flags error (master...0.13-consensus-flags-error) https://github.com/bitcoin/bitcoin/pull/9271 22:22 -!- fanquake [~fanquake@unaffiliated/fanquake] has quit [Quit: Leaving.] 22:24 -!- LeMiner [~LeMiner@unaffiliated/leminer] has quit [Read error: Connection reset by peer] 22:24 -!- LeMiner [~LeMiner@unaffiliated/leminer] has joined #bitcoin-core-dev 23:00 -!- dermoth [~thomas@dsl-66-36-132-180.mtl.aei.ca] has quit [Read error: Connection reset by peer] 23:01 -!- dermoth [~thomas@dsl-66-36-132-180.mtl.aei.ca] has joined #bitcoin-core-dev 23:32 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection] 23:33 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev 23:43 -!- cysm [cysm@unaffiliated/paracyst] has quit [Ping timeout: 260 seconds] 23:47 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has joined #bitcoin-core-dev 23:56 -!- Alina-malina [~Alina-mal@unaffiliated/alina-malina] has quit [Ping timeout: 256 seconds] 23:57 -!- Alina-malina [~Alina-mal@37.157.216.171] has joined #bitcoin-core-dev 23:59 -!- ThomasV [~ThomasV@unaffiliated/thomasv] has quit [Ping timeout: 256 seconds]