--- Day changed Fri Jul 01 2016
00:10 -!- Ginnarr [~Ginnarr@unaffiliated/ginnarr] has joined #bitcoin-core-dev
00:11 -!- owowo [~ovovo@unaffiliated/ovovo] has quit [Ping timeout: 240 seconds]
00:13 -!- cjcj [2e3b026a@gateway/web/freenode/ip.46.59.2.106] has joined #bitcoin-core-dev
00:52 < GitHub68> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/6a87eb0e4b47...da50997a3ee7
00:52 < GitHub68> bitcoin/master 0ce8e99 Wladimir J. van der Laan: windows: Add testnet link to installer
00:52 < GitHub68> bitcoin/master 975a41d Wladimir J. van der Laan: windows: Add testnet icon for testnet link...
00:52 < GitHub68> bitcoin/master da50997 Wladimir J. van der Laan: Merge #8285: windows: Add testnet link to installer...
00:52 < GitHub129> [bitcoin] laanwj closed pull request #8285: windows: Add testnet link to installer (master...2016_06_testnet_link_windows) https://github.com/bitcoin/bitcoin/pull/8285
00:55 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has joined #bitcoin-core-dev
01:10 < instagibbs> gmaxwell, yes the maximum ancestor/descendant stuff means that the transaction will be marked invalid and the funds deducted from your wallet until rescan :/
01:10 < instagibbs> err reindex or something
01:11 < gmaxwell> instagibbs: in this case the transaction when through after the parents confirmed.
01:11 < gmaxwell> the wallet rebroadcast accepted it the next time around and tada.
01:12 < gmaxwell> probably the coinselection process, when it considers 0conf inputs should only consider thoes with a low enough depth.
01:12 < gmaxwell> (perhaps two below the maximum to leave room for CPFP), and only if it's unable to meet that should it fall back to considering all inputs.
01:14 < instagibbs> gmaxwell, oh so even in failure it rebroadcasts?
01:14 < gmaxwell> yea, it saves it in the wallet before it tries to mempool it.
01:15 < gmaxwell> and since its saved, it keeps trying to rebroadcast.
01:15 <@wumpus> as long as the transaction is in the wallet it rebroadcasts
01:15 < gmaxwell> I dunno if the depth of an unconfirmed coin is easily discernable though from the wallet.
01:20 <@wumpus> not at is, it would need another (ugly) mempool dependency
01:20 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has quit [Ping timeout: 240 seconds]
01:21 < gmaxwell> bleh
01:22 <@wumpus> then again we already do an InMempool() check in IsTrusted, which determines whether the outputs of an transaction are considered spendable, so...
01:23 <@wumpus> if looking up the depth in the mempool is relatiely cheap that could be added too
01:24 <@wumpus> but yes I agree with the 'bleh' sentiment
01:24 <@wumpus> ideally mempool implementation details shouldn't matter to the wallet
01:25 <@wumpus> "if it cannot be submitted yet due to depth, try again later" seems a better approach
01:25 <@wumpus> the wallet already tries to avoid generating chains of unconfirmed
01:25 <@wumpus> so if it does, it really needs to
01:26 < gmaxwell> it doesn't really: so it tries to avoid spending unconfirmed coins, but if it must it treats short and long chains the same.
01:26 < gmaxwell> So in the example with the user I was supposting today, they had plenty of unconfirmed coins that were only 1 deep.
01:27 <@wumpus> yes I don't mean to imply that it looks at the mempool depth, it obviously doesn't
01:27 < gmaxwell> and one chain of 25.
01:27 <@wumpus> yes that would make sense - count *unconfirmed* depth
01:28 <@wumpus> then prefer coins that are shallow
01:28 < gmaxwell> it ended up that way because they had multiple unconfirmed few-bitcoin outputs, and then a 1 bitcoin output and were making many not huge payments... so it decided to keep reusing the change because it considered them all equal.
01:28 <@wumpus> that wouldn't require asking the mempool at all
01:28 < gmaxwell> yea, don't' really care about the mempool, the ismine tracing could count the depth until confirmed I guess.
01:29 <@wumpus> coins with a smaller unconfirmed depth could get some advantage in selection
01:29 <@wumpus> another thing in the long list of things that could be better in coin selection
01:31 < gmaxwell> well the way to do the advantage is the same way it prefers confirmed coins.
01:32 < gmaxwell> try the selection first with any too-deep coins excluded, and only if it fails attempt without that restriction.
01:32 <@wumpus> that's a possiblity, yes
01:33 -!- jtimon [~quassel@LAubervilliers-656-1-271-230.w193-248.abo.wanadoo.fr] has joined #bitcoin-core-dev
01:36 <@wumpus> though at some point the 'exclude these coins and do the selection again' list would become very long, and is less suited for non-boolean properties like input size (https://github.com/bitcoin/bitcoin/issues/7664)
01:36 <@wumpus> but sure, this could be bolted on too :)
01:37 < gmaxwell> well, it fits here since the failure results in an (temporarily) invalid transaction. ... just leaving them out is the right call for max depth, at least.
01:37 <@wumpus> I think at some point it would be nice to have a per-coin scoring system, then make the coin selection use that
01:38 < gmaxwell> yes, for some things, but a score for this should be infinite unless there is no other choice... since it guarentees the txn will not relay until its ancestors confirm some. :)
01:38 <@wumpus> but you make assumptions about other people's mempool depth
01:38 <@wumpus> I think a better approach would just be to *prefer* shallower transactions
01:38 <@wumpus> without an absolute theshold
01:39 < gmaxwell> They're not exclusive, prefer shorter but absolute threshold at the point where you won't relay the thing yourself anymore.
01:39 < gmaxwell> this usecase would actually have been better off with some kind of replacement, as they could have saved a good 20 transactions.
01:39 <@wumpus> and if it does manage to generates a temporarily invalid transaction, handle that more friendly, e.g. a warning instead of an error
01:41 <@wumpus> having the wallet have a hard dependency on a property of the mempool seems really brittle
01:42 <@wumpus> and it's not a solution that wallets that don't have such a close coupling to a node could use
01:42 < gmaxwell> I don't think this is really a mempool dependency.
01:42 -!- fengling [~fengling@58.135.95.131] has joined #bitcoin-core-dev
01:42 <@wumpus> in principle it's a matter of optimizing the speed at which the transaction can be submitted, longer chains result in slower confirmation times
01:42 <@wumpus> very long chains even in temporary rejection
01:42 < gmaxwell> If it's spending here the transaction is ismine, which means all unconfirmed ancestors are wallet txn.
01:43 < gmaxwell> So it just needs to know the longest depth.
01:43 <@wumpus> but long chains, even if they are accepted, are not a good thing in itself
01:44 < gmaxwell> (er not IsMine, but IsFromMe)
01:44 -!- jannes [~jannes@178.132.211.90] has joined #bitcoin-core-dev
01:46 < gmaxwell> wumpus: they're not that bad, esp with ancestor feerate mining; and relay being not totally broken for them in 0.13.
01:46 <@wumpus> but I hope you agree that they are worse than shallow transaction chains
01:46 -!- fengling [~fengling@58.135.95.131] has quit [Ping timeout: 240 seconds]
01:46 <@wumpus> 'not that bad' yes there are always much worse things :)
01:46 < gmaxwell> all things equal... but compared to a choice that is less private or pays more fees? 
01:47 <@wumpus> well the individual scoring weights would have to be determined, yes, or even depend on some setting
01:47 <@wumpus> paying more fees may be ok if it means the transaction has a larger chance of being picked
01:47 < gmaxwell> because we never split large change output, a lot of usage patterns result in no real choice in any case.
01:48 <@wumpus> less private, well long transaction chains are a great way to say 'hey this is a wallet sending out transactions serially'
01:49 < gmaxwell> sure, but often better to just respend change than join a dozen otherwise unrelated inputs. 
01:51 <@wumpus> if it requires joining them, yes I agree
01:52  * wumpus wishes someone would do a serious study about what wallet behavior would make sense
01:53 -!- AaronvanW [~ewout@unaffiliated/aaronvanw] has quit [Read error: Connection reset by peer]
01:53 -!- fengling [~fengling@58.135.95.131] has joined #bitcoin-core-dev
01:55 < gmaxwell> Would we do anything with it?... a while back someone proposed a change (to remove extranious inputs), I suggested that it might result in wallets grinding down coins into small amouts more often. He made a simulator that showed it would, then we took the change. Then later users showed up complaining about the wallet grinding down inputs when they didn't use to... 
01:56 -!- AaronvanW [~ewout@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
01:56 <@wumpus> well the problem is that we're too busy running from issue to issue to look at a higher level
01:56 <@wumpus> well at least I am
01:56 < gmaxwell> right. I agree with that.
01:57 < gmaxwell> just collectively we are.
01:57 <@wumpus> so now long chains are an issue, long chains are fixed by adding yet another special case, but without considering impact on other things
01:57 < gmaxwell> You're right that someone looking at it more holistically would be good, part of the problem in that issue was that even there it was only asking a very narrow question.
01:57 <@wumpus> at some point I just worry all those hacks make things worse, instead of picking some simple algorithm that does fine in most cases
01:58 < gmaxwell> I don't really think there is much to consider on the subject of "avoid going over your own maximum chain depth if at all possible".
01:58 <@wumpus> but I don't know, I don't have the ovierview
01:58 < gmaxwell> since producing a txn even you won't relay when you could have avoided it seems clearly wrong.
01:58 < phantomcircuit> gmaxwell: the wallet should probably have all of the unconfirmed depdencies of a transaction
01:59 <@wumpus> so dropping of extraneous inputs was bad?
01:59 < phantomcircuit> and then remove the dependencies after some high number of confirms which aren't relevant
02:00 < gmaxwell> phantomcircuit: in these cases all the unconfirmed dependencies will be wallet transactions-- IsFromMe otherwise it won't spend the coin!
02:00 <@wumpus> normally you'd say that choosing a minimum set that covers the value to be spent would be optimal
02:00 < phantomcircuit> but yeah you're right it has to pass IsMine which means the wallet already has all the info you need to calculate depth with just coins view
02:00 <@wumpus> but yes, I'm sure there are lots of other contraints and scores that should be taken in to account
02:00 < phantomcircuit> reading top down so my comments might be already worked out :P
02:00 < gmaxwell> wumpus: the result ends up being the smallest possible change. so it breaks your wallet into lots of tiny little coins.
02:00 <@wumpus> which was my point before
02:00 -!- molly [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
02:01 <@wumpus> so why wasn't that reverted?
02:02 < gmaxwell> I don't know/understand why it wasn't.
02:02 < gmaxwell> I think when it went in there was some expectation that further improvements would come, and they didn't. Then it was released. Then people showed up and complaints (opening an issue) and we figured out that the behavior change was due to removing extranious inputs.. and then?  I don't know
02:02 <@wumpus> we just have too much on our plate
02:02 < gmaxwell> then something else caught fire.
02:03 <@wumpus> sometimes I really want to run around screaming with my hands on my head
02:03 < phantomcircuit> wumpus: that sounds like fun
02:03 <@wumpus> it's not possible to handle this all anymore
02:04 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 250 seconds]
02:04 <@wumpus> we really need someone that focuses on wallet improvements
02:04 < gmaxwell> (privacy wants you to spend all payments to a single address at once, so that you don't get a rolling linkage that eventually cross taints every address in your wallet--- so maybe just attempting to do this would counteract most of the grinding. thats the kind of thing you want tne step back and overview to answer)
02:05 -!- Retric [~Retric@ppp118-208-195-193.lns20.hba1.internode.on.net] has quit [Quit: Leaving]
02:06 <@wumpus> someone that has an overview of what the heck is happening in the wallet
02:06 < gmaxwell> There are a number of positive wallet things going on, at least. But there is a lot of time spent twiddling things in the weeds rather than setting up prioties and identifying larger scale issues.
02:06 <@wumpus> but how to sort 'bad' improvements from good ones?
02:06 <@wumpus> I don't know anymore
02:07 < phantomcircuit> i've been trying to improve the wallet
02:07 <@wumpus> yes, thanks for that
02:08 < phantomcircuit> unfortunately yeah it's a lot of running around in the weeds trying to fix things up... to make it easier to do bigger things
02:08 <@wumpus> all in all the utxo/coin based approach does cause a lot of non-trivial difficulties, both at the node (utxo set sprawl) level as in the wallet
02:08 < gmaxwell> in any case, I didn't bring any of this up here to complain-- I brought it up because initially I couldn't figure out the cause, and then updated when I did.
02:09 < gmaxwell> it avoids a lot of replay problems however, which avoids other kinds of sprawl
02:09 <@wumpus> no, I'm not trying to complain either, but sometimes I just don't know anymroe
02:10 -!- fengling [~fengling@58.135.95.131] has quit [Ping timeout: 240 seconds]
02:10 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev
02:10 < gmaxwell> well, from another perspective-- we have no deadline. These same issues existed in 2011 (actually, much worse), but it didn't matter because hardly anyone used the system. :)
02:11 <@wumpus> but that's part of what worries me, yes these same issues existed in 2011
02:11 <@wumpus> no, no deadline, but e.g. the utxo growth is worrying
02:12 <@wumpus> we're running against (soft) walls
02:12 < gmaxwell> yes, I'm concerned about utxo growth.
02:12 < gmaxwell> I happened to chart data from all those reindexing tests I just did, and the time to verify a block is increasing (for the same amount of txn data).
02:13 <@wumpus> and all the while joe user is complaining about lack of *scaling*, while the system is already seemingly bursting at its seems
02:13 < gmaxwell> I was theorizing that this was from polylog behavior in the database and worrying, but phantomcircuit gave an alternative argument that the reduction in spammy transactions relative to non-spammy ones may be resulting in lower cache hitrates.
02:13 < gmaxwell> so I'm going to try to figure out if thats the case.
02:14 < gmaxwell> (by increasing, I mean increasing enough that it was clearly visible on a last-8064 block plot)
02:15 <@wumpus> "the time to verify a block is increasing (for the same amount of txn data" yes, indeed, I also intended to plot time per block, but that much was clear just from looking at timestamps :(
02:16 < gmaxwell> well the difference between 1MB and smaller blocks was expected, but the increase still existed when looking at only large blocks.
02:16 <@wumpus> yes
02:16 <@wumpus> recent blocks verify slower
02:17 <@wumpus> even of the same size
02:17 <@wumpus> so *let alone* what bigger blocks would do
02:17 < gmaxwell> At least segwit doesn't increase the worst case amount of utxo growth or accessing...
02:18 -!- fengling [~fengling@58.135.95.133] has joined #bitcoin-core-dev
02:19 -!- xiangfu [~xiangfu@58.135.95.135] has quit [Ping timeout: 250 seconds]
02:30 -!- Ginnarr [~Ginnarr@unaffiliated/ginnarr] has quit [Quit: Textual IRC Client: www.textualapp.com]
02:38 -!- owowo [~ovovo@185.65.134.81] has joined #bitcoin-core-dev
02:38 -!- owowo [~ovovo@185.65.134.81] has quit [Changing host]
02:38 -!- owowo [~ovovo@unaffiliated/ovovo] has joined #bitcoin-core-dev
02:42 -!- pedrobranco [~pedrobran@79.242.108.93.rev.vodafone.pt] has joined #bitcoin-core-dev
02:56 < pedrobranco> sipa: hi, when you have the time can you help on the doubts in https://github.com/bitcoin/bitcoin/pull/7551? Thanks :)
03:04 < sipa> pedrobranco: i answered quickly
03:04 < sipa> pedrobranco: feel free to disagree by the way... if my view is not obvious here, perhaps the semantics should be different and more clear
03:09 < pedrobranco> sipa: thanks.
03:35 -!- fengling [~fengling@58.135.95.133] has quit [Quit: WeeChat 1.4]
03:37 <@wumpus> wow, that wallet drop-unnecessary-inputs-from-best-subset change had been open from sep 2014
03:38 <@wumpus> https://github.com/bitcoin/bitcoin/pull/4906
03:40 <@wumpus> it has not yet made it into any release
03:41 < MarcoFalke> it has
03:41 < MarcoFalke> it was backported to .12
03:41 <@wumpus> fuck
03:41 <@wumpus> yes I see: 96e8d120336cf4312cd5f42ba2f9aff17d4ad414
03:42 <@wumpus> that was my own stupid idea probably
03:43 < MarcoFalke> Up to now I have not seen proof that this made things considerably worse
03:44 < MarcoFalke> Someone should come up with a framwork that mimics common wallet use cases
03:44 <@wumpus> ok
03:44 < MarcoFalke> e.g 'exchange': something with a huge in-out volume/count
03:44 < MarcoFalke> then a "normal user": always waits for confirmations, low volume
03:45 <@wumpus> right, that would make sense, for a project taking the wallet part seriously
03:45 < MarcoFalke> and then different behaviors like 'always send small coins' (pay for coffee each day)
03:45 -!- jtimon [~quassel@LAubervilliers-656-1-271-230.w193-248.abo.wanadoo.fr] has quit [Remote host closed the connection]
03:46 < MarcoFalke> and 'receive large coins' (get payed in btc)
03:46 < MarcoFalke> etc
03:47 <@wumpus> right.
03:49 <@wumpus> gmaxwell mentioned above that he did see some reports of worse behavior after the change
03:49 <@wumpus> now it's not clear whether to revert it or not
03:50 < gmaxwell> We recieved an issue with a business with a large wallet reporting specifically the expected behavior there.
03:50 <@wumpus> why don't businesses with large wallets never help with the wallet development?
03:51 <@wumpus> it seems it would be in their best interest
03:51 < gmaxwell> because there are very few left, most are using fully hosted things that have custom software.
03:51 <@wumpus> but I don't think this was even reported on github
03:51 < gmaxwell> I thought it was.
03:51 <@wumpus> it's just frustrating
03:51 < MarcoFalke> it was
03:51 <@wumpus> oh?
03:51 < gmaxwell> And utxo growth coincided with it. This was previously discussed in meetings IIRC and then ... I'm not sure what happened. 
03:52 <@wumpus> that's not important. What is important is what to do now
03:52 <@wumpus> revert it?
03:52 < MarcoFalke> #7664 #7657
03:53 < gmaxwell> maybe it was dropped for good reason. I just can't recall.
03:54 < gmaxwell> At the end of the day, the pruning itself was correcting a bug. The potential issue is that the bug was covering up that the non-bugged behavior is bad (and the simulations showed that it, as I said, would grind down wallets into lots of tiny coins)
03:54 < MarcoFalke> Imo it shouldn't matter if we revert it or not. I haven't yet seen simulations which prove that either would place an advatage/disatvantage
03:54 < gmaxwell> MarcoFalke: IIRC there were simulations posted that showed that the behavior caused far more utxo.
03:55 < MarcoFalke> No one verified that the simulations were implemented according to the current coin selection code
03:55 < MarcoFalke> I have seen single satoshi outputs in those simulations
03:55 < MarcoFalke> which does not happen with our code
03:55 < MarcoFalke> Also the author found a bug in the implementation about two weeks ago or so
03:55 < gmaxwell> The result is also intutive to me. It results in making the smallest possible outputs. This is pessimal when you consider that avoiding utxo size wants, for a given users balance, the largest value outputs. 
03:56 < MarcoFalke> Ideally, we'd implement the wallet benchmark framework in cpp and have it just use our wallet code?
03:57 <@wumpus> if it would be modular enough for that :)
03:57 <@wumpus> possibly separate the coin selection logic out to a separate file, which could also be used by the simulation framework
03:58 <@wumpus> it could use an abstraction of a list of coins with some properties instead of a wallet as input
03:58 <@wumpus> it would make some things a lot easier, like trying out what the algo does in certain circumstances, without actually having to fake a wallet
03:59 < gmaxwell> An even better behavior would be to add all other inputs to the same addresses being spent, up to some limit to prevent very large transactions, to sweep them up. This is a privacy improving strategy, as well as a fee minimizing strategy under an assumption that fees will increase in the future.   My guess is that I didn't NO DONT the pruning change because in and of itself it was right, and the
03:59 < gmaxwell>  grinding should be fixed by something like this.
04:00 < gmaxwell> but it went off my radar, also changes to coinselection are a pain because a bunch of th tests freeking hard code the behavior.
04:00 <@wumpus> if that was the only thing making changes to coin selection a pain :)
04:01 <@wumpus> it's just almost impossible to agree what is better behavior, which is what talled #4906 for so long and resulted in it eventually being merged, probably wrongly
04:02 <@wumpus> so a) it should be better behavior b) it needs to be implemented correctly, and indeed, we have no good tests or simulation framework
04:04 <@wumpus> sweeping up as many as possible inputs to the same address would make sense
04:04 <@wumpus> there is no privacy benefit from having more spends to one address
04:04 <@wumpus> then again it assumes bad-case behavior from the user, reusing addresses
04:05 < gmaxwell> I don't want to be hard on you, but Xekyo ran simulations which showed 25% to 400% increase in the UTXO set size under some simulation loads. 
04:05 <@wumpus> I think it would make sense to design a standard interface for wallet coin selection algorithms, separately from any sepecific wallet
04:05 < gmaxwell> We had simulations, they might not have been ideal.
04:05 <@wumpus> so that research and simuation can be done outside of the specific scope of bitcoin core
04:06 < gmaxwell> They showed bad behavior from this change. From intution I predited specifically this bad behavior and asked for the simulations.  Our issue is not that we need more simulations.
04:06 < gmaxwell> (well we do, but that didn't help here!)
04:06 <@wumpus> ok, never mind then
04:06 < gmaxwell> and still MarcoFalke is not agreeing that there is a potential issue.
04:06 <@wumpus> just trying to think of some things that might help, out of the blue
04:06 < gmaxwell> yea.
04:06 <@wumpus> if it's all hopeless I'll just shut up
04:07 < gmaxwell> xekyo also identifyied a useful strategy, making the coinselection target double the value instead of the value. This results in change of useful sizes. 
04:07 <@wumpus> *more* simulations may not be a solution, but better, correctly implemented ones would
04:08 <@wumpus> if the answer of this is up to whether the simulation is implemented correctly, then we're not really helped a bit, I agree
04:09 < gmaxwell> re: assumes bad-case behavior from the user, well, indeed, it's no effect if the user doesn't reuse. But no harm, and reuse is ubiquitious. The (vast?) majority of bitcoins in circulation are held in reused addresses.
04:09 <@wumpus> the vast majority of those may not  be using bitcoin core in the first place
04:10 <@wumpus> many wallets encourage address reuse, or can only do address resuse
04:10 <@wumpus> that rule may help them a lot :)
04:11 < gmaxwell> pretty much every active user has some reuse though, consider that thing that sends tips for commits.
04:11 <@wumpus> anyhow that speaks too of generalizing the coin selection algorithm beyond the specific software level
04:11 < gmaxwell> means that all of us have reuse, even if we otherwise act perfectly ourselves. :)
04:11 <@wumpus> yes, that is true, ideally that should use some BIP32 construction
04:12 <@wumpus> to be honest I always use manual coin selection
04:12 <@wumpus> (apart from testing)
04:13 < gmaxwell> I do too. (and always spend all the coins connected to an address, when I spend any)
04:14 <@wumpus> and that hould ideally coincide with changing the donation address
04:14 < gmaxwell> they've gotta get spent someday or the coins are lost, fees are likely lower now than in the future... and spending at once avoids privacy harm from constantly interlinking inside the wallet.
04:14 < gmaxwell> yea, I do that with my bct address. change the one on the site, when I spend all the coins.
04:15 < gmaxwell> for that tip commit thing, I just haven't been spending its payments, I think... as I dunno how to change it.
04:16 <@wumpus> they have a little web interface where you can log in and change the address IIRC
04:18 < gmaxwell> I wonder what the average coin value is in the utxo set and how thats evolved over time. (it's not quite as simple as the set size evolution over time, since more coins have also been introduced)
04:18 -!- murch [~murch@p4FE3A233.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
04:19 < gmaxwell>   "errors": "WARNING: abnormally high number of blocks generated, 4477 blocks received in the last 4 hours (24 expected)"
04:19 < gmaxwell> lol
04:20 <@wumpus> yes that would be an interesting utxo statistic
04:20 <@wumpus> gmaxwell: https://github.com/bitcoin/bitcoin/pull/8275 :p
04:27 < GitHub78> [bitcoin] laanwj opened pull request #8298: wallet: Revert input selection post-pruning (master...2016_06_revert_wallet_input_postprune) https://github.com/bitcoin/bitcoin/pull/8298
04:33 -!- molly is now known as moli
04:37 < sturles> I can see some HD related pull requests with no milestone set.  Any chance of HD support in 0.13, or is it 0.14 material?
04:37 <@wumpus> basic HD support has been merged for 0.13
04:38 <@wumpus> the feature freeze for 0.13 is long past now though, so everything else is for 0.14 soonest
04:38 <@wumpus> feel free to help testing them though, that helps a lot
04:38 <@wumpus> if anything there's a lack of involvement and testing and review for wallet PRs, especially UI oriented ones
04:39 < gmaxwell> we certantly had wallet features slip due to lack of love.
04:40 -!- Prattler [~ttttt@78-60-12-33.static.zebra.lt] has quit [Read error: Connection reset by peer]
04:42 < sturles> I haven't used the UI for years..  Will start testing HD stuff.
04:45 -!- blur3d [~blur3d@49.187.28.114] has joined #bitcoin-core-dev
04:46 -!- cryptapus_afk is now known as cryptapus
04:51 <@wumpus> at least for BIP32, in contrast to coin selection, the goal is well-defined and delimited :)
04:56 < murch> wumpus: I'm working on the latter. :)
04:57 <@wumpus> murch: awesome!
04:58 < murch> I'm writing a Master thesis on Coin Selection. I've been talking about that to some people here before.
04:58 < murch> (Pieter, Greg, Greg, Marco)
04:59 <@wumpus> great, some more general reserach there would be very welcome
04:59 < murch> One focus will be trying to analyze goals, prerequisites and constraints, then the results of varying approaches for selection
05:00 <@wumpus> yea exactly, what I posted here https://github.com/bitcoin/bitcoin/pull/8298#issuecomment-229928583  I think what is needed first is a list of criteria to judge coin selection algorithms by
05:00 <@wumpus> right now we can't tell up from down
05:01 <@wumpus> (unless it is obvious breakage)
05:03 < murch> Thanks, I hadn't seen that one yet.
05:04 < murch> I was actually the one that provided the simulation in 4096
05:05 <@wumpus> I mean there are obvious concerns such as CPU usage and memory usage, fee minimization, but also more 'tragedy of the commons' issues such as privacy concerns, utxo growth concerns (though that also coincides with performance a bit, keeping the wallet's utxo set small keeps the global set also smaller)
05:06 < murch> Actually, I think that the pruning should have had little effect, as there should only be anything to prune when a second pass is needed. Otherwise, since the last added UTXO would be the smallest, there should not be any UTXO prunable.
05:06 <@wumpus> it has very little effect
05:06 <@wumpus> (but apparantly visible in some cases)
05:07 -!- justanotheruser is now known as UserNoJustanuu
05:07 -!- UserNoJustanuu is now known as justanotheruser
05:08 <@wumpus> though of course it's not entirely certain whether the reported problems were due to this change, or another change, or a change in the general usage of bitcoin not directly related to a change in our wallet
05:08 <@wumpus> or a combination of factors including this change
05:09 <@wumpus> in any case, if it has 'little effect' that's also enough reason to revert, I think. If it achieves hardly anything good, it shouldn't have been changed.
05:10 < murch> wumpus: Completely agree.
05:11 < murch> Well, I hope that I'll be able to provide some real improvements in the following months. :)
05:12 < murch> Although I was surprised how well the current implementation does. It's not trivial to improve on it, and not make it deterministic in some fashion. :)
05:13 < murch> Oh, and providing some metrics to compare Coin Selection approaches is the focus of the work
05:13 <@wumpus> I hope so too!
05:14 <@wumpus> good to hear the current approach is fairly good, on the other hand, that is going to make it even harder to replace by something better :)
05:14 <@wumpus> I think the most common complaints are that it does bad with very large wallets, or containing lots of small inputs
05:14 < murch> By the way, wumpus, do you know of any other wallet usage data? I got the moneypot.com wallet from #4096, but more good testcases would be grand.
05:14 <@wumpus> apart from that it's not something people usually stumble upon
05:15 <@wumpus> no, unfortunately not
05:16 <@wumpus> I don't have any realistic big wallets
05:16 < murch> Okay, too bad. :)
05:16 <@wumpus> (this is also an issue which has existed from 2011, which affects both coin selection and general wallet scaling work, people with big wallets are very reluctant to share them even with developers :-) )
05:17 < murch> Yeah, I'd imagine. :-/
05:18 <@wumpus> which is also why I always encourage companies which have to cope with them to get involved in development themselves, that avoids having to share anything with third parties
05:18 < murch> The code in wallet.cpp is pretty hard to understand, too. I'd love to refactor it, maybe even to just understand it properly…
05:18 <@wumpus> phantomcircuit is working on that too
05:18 < murch> oh really?!
05:19 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
05:19 < murch> mh, just refactoring it, or also improving?
05:19 <@wumpus> in a way it's kind of a chicken and egg problem though, people rarely dare to change things *becuase* they're not sure how they work exactly, but evaluating refactors is also made hard by that because it's non-trivial to see that behavior stays the same
05:20 <@wumpus> both, but not coin selection AFAIK
05:20 < murch> Yeah, on the other hand it's in a state where it is really hard to add unittests to check whether the behavior remains unchanged.
05:20 < murch> okay, excellent
05:20 <@wumpus> yes, that too.
05:23 <@wumpus> and years of proposals of doing things differently have made me think that it being hard to understand is not so much a result of the organizatin of wallet.cpp, but that the underlying subject matter is hard
05:24 <@wumpus> people tend to read the code and then blame that the code is difficult, but it's not entirely clear whether it is *unnecessarily* difficult
05:25 <@wumpus> it's clearly a complex problem too
05:25 <@wumpus> maybe it could be divided up into better manageb;e sub-problems though
05:25 <@wumpus> manageable*
05:27 <@wumpus> the thing is, before wallet.cpp was written, cryptocurrency wallets was a problem no one ever considered before, it was necessarily ad-hoc
05:28 <@wumpus> this is very different from say, a web server, where everyone knows what is expected from it, how it usually should be structured, and so on
05:30 < murch> wumpus: It's surely complicated, but there are some methods that could be extract, and some things are just somewhat obfuscated by very brief variable names. E.g. I've been looking at how and when the fee gets decided for the transaction and it's still baffling me.
05:30 <@wumpus> so the refactor is also a discovery process, learning how to best structure this novel application
05:32 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.]
05:32 <@wumpus> well yes it's complicated - but does it need to be complicated, is there a less complex way that would be just as good ,or better? would that simpler way still satisfy the requirements? (and in some cases: what are the requirements, even?)
05:34 < murch> wumpus: I think it could be more comprehensible if key management and coin selection were separated into different classe for example. They are pretty far apart for being part of one class.
05:34 <@wumpus> things like variable names are trivialities, sure, on first reading of the code it helps to have better names, but once you learn what they are it doesn't really matter what they are anymore (e.g. mathematicians do fine with names such as a b c :)
05:34 < murch> Most of Coin Selection could actually be completely static, as it basically is just reliant on the spending target, the UTXO set and the desired fee level.
05:34 <@wumpus> yes coin selection is a clear unit
05:35 <@wumpus> I think it would be nice to factor it out to a separate unit with a separate interface, that just gets the information that it needs and returns the selected list
05:35 <@wumpus> this would also be useful for simulations
05:35 < murch> well, wallet.cpp is just a bit of a moloch to delve into at 3500 LOC. ;)
05:35 < murch> wumpus: exactly
05:35 <@wumpus> I've seen so much worse at some companies I've worked at :-)
05:36 < murch> Also, adding, watching and tracking UTXO could probably be separated off
05:36 <@wumpus> yes, that would make sense
05:37 <@wumpus> did you see https://github.com/bitcoin/bitcoin/pull/7823 ?
05:38 < murch> nope, but I just subscribed.
05:38 < murch> I am not nearly as familiar with the github repository as I'd like. :-/
05:38 <@wumpus> the idea is quite nice, but it's *hard* to see what its interaction with different things would be, such as reorganizations and conflicts
05:38 < murch> I should probably check out all issues tagged with "wallet" soon.
05:38 <@wumpus> yes :)
05:39 <@wumpus> in any case, coin selection in itself is enough of a subject to fill a master's thesis I think, I'd warn against scope creep, or trying to fix the world at once :)
05:39 -!- davec_ [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Read error: Connection reset by peer]
05:39 < murch> hehe
05:40 -!- davec_ [~davec@cpe-24-243-251-52.hot.res.rr.com] has joined #bitcoin-core-dev
05:41 < murch> yeah, it's a bit overwhelming at times. I've been reading about Subset Sum solvers the past days, and delving through wallet.cpp to understand how fees are handled.
05:42 < murch> Anyway, back to work. ;) TTYL
05:43 <@wumpus> later :) hope you manage to make head and tail of it
05:43 <@wumpus> hm the new option -blockmaxcost is going to give user understanding problems: https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L455
05:43 <@wumpus> should this be a hidden/debug option? if not, it should be documented better
05:53 -!- TheFactory7 [uid164731@gateway/web/irccloud.com/x-jxaukbigjfvzpkje] has joined #bitcoin-core-dev
06:06 -!- Prattler [~ttttt@78-60-12-33.static.zebra.lt] has joined #bitcoin-core-dev
06:10 -!- Prattler [~ttttt@78-60-12-33.static.zebra.lt] has quit [Client Quit]
06:18 < jonasschnelli> sipa, wumpus: Do I get this right: -reindex does re-create the block index and recreates the utxo set (including all signature validation)    while -reindex-chainstate will only recreate the utxo set with the current block index?
06:20 < sdaftuar> wumpus: -blockmaxcost is supposed to be the recommended way to configure the mining code, going forward
06:21 < sdaftuar> wumpus: and -blockmaxsize is in the we-want-to-deprecate-in-the-future category
06:21 < sdaftuar> (the mining code is optimizing for fee-per-block-cost, not fee-per-serialized-byte)
06:22 < sdaftuar> but yes, we need to document all this before release
06:22 < sdaftuar> heh, yeah that --help message text is not very informative
06:27 -!- Retric [~Retric@ppp118-208-219-199.lns20.hba1.internode.on.net] has joined #bitcoin-core-dev
06:29 -!- tadasv [ttttt@gateway/shell/panicbnc/x-unhqyribepejyjop] has joined #bitcoin-core-dev
06:30 <@wumpus> sdaftuar: a translator was wondering what kind of cost this was, I think they assume that it's something with fee
06:31 <@wumpus> jonasschnelli: correct
06:31 < sdaftuar> oops.  yeah that is bad language
06:31 -!- zooko [~user@2601:281:8000:8387:5157:31b2:4a62:d545] has joined #bitcoin-core-dev
06:31 < sdaftuar> what's the timeline for improving that text, is it too late for making changes now that affect translation?
06:32 < sdaftuar> i guess we can just document in the release notes?
06:32 <@wumpus> well given that translators are confused by it (and hence unable to translate it effectiely) I wouldn't mind changing the message
06:33 <@wumpus> there also needs to be mention in the release notes, but release notes aren't really documentation, just 'news'
06:33 <@wumpus> e.g. you wouldn't assume that someone that is looking for documentation for an option to go through all previous release notes to find it documented
06:33 < sdaftuar> a simple change might just be to reference the BIP where it's defined; that wouldn't necessarily impose an additional burden on translators if it's just "(BIP 141)" or something at the end of it
06:34 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 240 seconds]
06:34 <@wumpus> what needs to be made clear is that this is just an abstract cost
06:34 <@wumpus> referencing the BIp would make sense, yes
06:34 <@wumpus> in any case, even if it's too late for this to be translated, a better english documentation message would go further than a confusing translated one :)
06:35 < sdaftuar> agreed!
06:35 < sdaftuar> perhaps block cost shouldn't be translated at all actually, if we're referencing the BIP
06:39 < sdaftuar> i updated #8294 so we don't forget this
06:43 < MarcoFalke> gmaxwell: I don't think lack of progress in improving of coinselection is due to the test hardcoding the behavior.
06:43 < MarcoFalke> Right now they save us from introducing accidental regressions, which is nice
06:44 < MarcoFalke> If someone comes up with a new idea, the unit test need to go anyway and will be replaced by new ones, so not really an issue
06:45 < MarcoFalke> Also the 25% to 400% performance loss may be flawed, as the coin generator was not adjusted to what happens in the real network
06:45 < MarcoFalke> (re simulation)
06:47 < MarcoFalke> I remember there was a site which put every address which was ever in a wallet. (Combining whenever two addresses happen to be used in the inputs of the same tx)
06:47 < MarcoFalke> This could be useful to verify coin generators in simulations are doing their job properly
06:48 -!- cjcj [2e3b026a@gateway/web/freenode/ip.46.59.2.106] has quit [Ping timeout: 250 seconds]
06:51 -!- JackH [~Jack@79-73-186-51.dynamic.dsl.as9105.com] has quit [Quit: Leaving]
07:09 -!- blur3d [~blur3d@49.187.28.114] has quit [Quit: blur3d]
07:13 -!- Retric [~Retric@ppp118-208-219-199.lns20.hba1.internode.on.net] has quit [Quit: Leaving]
07:18 -!- cjcj [2e3b026a@gateway/web/freenode/ip.46.59.2.106] has joined #bitcoin-core-dev
07:22 -!- Cats2 [~CattieCat@2607:fcc8:9e44:2500:dd5f:8b0e:1991:ee42] has joined #bitcoin-core-dev
07:23 -!- zooko [~user@2601:281:8000:8387:5157:31b2:4a62:d545] has quit [Ping timeout: 250 seconds]
07:25 -!- ClockCat [~CattieCat@2607:fcc8:9e44:2500:dd5f:8b0e:1991:ee42] has quit [Ping timeout: 250 seconds]
07:26 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has quit [Ping timeout: 250 seconds]
07:28 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has joined #bitcoin-core-dev
07:28 -!- pedrobranco [~pedrobran@79.242.108.93.rev.vodafone.pt] has quit [Remote host closed the connection]
07:34 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has left #bitcoin-core-dev []
07:38 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
07:38 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Client Quit]
07:39 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
07:53 -!- pedrobranco [~pedrobran@79.242.108.93.rev.vodafone.pt] has joined #bitcoin-core-dev
08:00 < morcos> wumpus: gmaxwell: i don’t feel strongly enough to make a big argument about it, but if it was up to me i wouldn’t bother reverting 4906.  I agree that we didn’t have sufficient justification to merge it in the first place, but we already crossed that bridge, and discussed it after the fact (more than once).   I’m not sure we’re confident enough that its clearly worse to risk making the same mistake in the other directi
08:01 < morcos> direction by reverting it
08:01 < morcos> I suppose my point is I’d err on the side of being conservative with changes to coin selection and only making them when someone has put in the effort to really study them.  Since 0.12 has been out for a while running with 4906, it now seems like a change to me to revert it.
08:05 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 276 seconds]
08:06 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev
08:14 < jonasschnelli> sipa: A full IBD (against random peers) with current master took ~3h4min, a -reindex-chainstate took ~1h10min
08:15 < jonasschnelli> http://bitcoinsrv.jonasschnelli.ch/log.reindex-chainstate.nodebug.da5099.txt, http://bitcoinsrv.jonasschnelli.ch/log.reindex-chainstate.nodebug.da5099.full.txt
08:15 < jonasschnelli> IBD: http://bitcoinsrv.jonasschnelli.ch/log.ibd.nodebug.da5099.txt, http://bitcoinsrv.jonasschnelli.ch/log.ibd.nodebug.da5099.full.txt
08:15 < jonasschnelli> now doing a full -reindex
08:15 < jonasschnelli> Used -dbcache=8000 for both runs
08:37 <@wumpus> morcos: I don't think 'we crossed that bridge' is a good argument, if this was no improvement, it should not have been merged, and it should be reverted (it should have been reverted a long time ago!)
08:37 <@wumpus> morcos: I don't see any reason to keep the code if it i not an improvement
08:37 <@wumpus> and if we should be conservative about changes to coin selection, which we should have been in the first place, then again we shouldn't have this change without understanding it
08:38 < morcos> wumpus: ok like i said i won't argue, it just makes me nervous that somehow reverting could be worse (interaction with something else that changed that we're not considering), but i don't have any reason to believe that
08:38 <@wumpus> I don't think having the mistake out in 0.12 is  a reason to keep it now
08:38 <@wumpus> don't get attached to your mistakes :)
08:38 < morcos> i won't get attached to somebody's mistakes  :)
08:39 <@wumpus> then again I don't feel strongly about it either, but I'm tired of it coming up every time
08:39 <@wumpus> let's just make a damn decision about it
08:39 <@wumpus> I'm sure if we decide not to revert it now, then someone will bring it up again after a month or so
08:39 <@wumpus> I don't want this following me around forever
08:41 <@wumpus> I don't see how this could interact with anything else
08:44 < bsm1175321> Woah.  I just built github master and on testnet something has gone horribly wrong: I have a negative balance!
08:45 < morcos> bsm1175321: checked how?
08:45 < bsm1175321> |mcelrath@jane:~> bitcoin-cli -testnet getbalance ''
08:45 < bsm1175321> -56.85287200
08:45 < sipa> drop the ''
08:45 < bsm1175321> It seems to have suddenly fixed itself.
08:45 < sipa> you're computing an account balance
08:45 < sipa> not the wallet balance
08:45 <@wumpus> you used to wrong way to query your balance! woe you!
08:46 < bsm1175321> Without the quotes I got:
08:46 < bsm1175321> |mcelrath@jane:~> bitcoin-cli -testnet getbalance 
08:46 < bsm1175321> 0.00000000
08:46 < morcos> i hesitate to ask this question, but can we finally remove accounts for 0.14
08:46 <@wumpus> (we're crazy that there are so many, including wrong ways to query your balance)
08:46 < morcos> talk about having an issue follow us around
08:46 <@wumpus> morcos: I tried to push for a label API to replace it
08:46 < bsm1175321> Which is also wrong.  A few minutes later it seems to have synced up...and is now showing correct balances.  What happened?
08:46 < sipa> there was a 4000 deep reorg
08:46 <@wumpus> morcos: unnecessary to say, it didn't get enough attention, and is still lingering, I hope to pick it up again for 0.14
08:47 < bsm1175321> +1 for getting rid of accounts...
08:47 <@wumpus> morcos: https://github.com/bitcoin/bitcoin/pull/7729   I think the release after that we can really remove the account functionality
08:47 < morcos> wumpus: ok 3rd topic. i thought i head you guys mention that something has gotten slower recently.  is it just reindex, or actually something with connecting blocks?
08:48 < bsm1175321> Next question: I need to fund transactions and be sure that their txid is not malleable.  I was hoping 'fundrawtransaction' had an option to take only segwit inputs, but it seems it does not.  What's the best way to achieve this?
08:48 <@wumpus> morcos: AFAIK the summary there was: gmaxwell accidentally -txindex
08:48 < sipa> morcos: and nobody recently tested -reindex with default -dbcache
08:48 < morcos> wumpus: oh i definitely missed that conclusion. :)  
08:48 <@wumpus> morcos: yes, more recent blocks are slower to validate, comapred to older blocks of the same size - but this isn't a reversion in the code
08:49 < morcos> wumpus: wait, yes its that last thing i'm asking about
08:49 < morcos> why is that?
08:49 <@wumpus> it's likely a by-product of increasing utxo size
08:50 < morcos> and that affecting cache hit rate?
08:50 <@wumpus> and the utxo database is reaching the size that leveldb can handle w/ good performance, e.g. now lots of crazy seeking and reading all over your disk, can't be good for performance
08:50 < morcos> oh
08:50 <@wumpus> yes that was another hypothesis
08:50 <@wumpus> <gmaxwell> I was theorizing that this was from polylog behavior in the database and worrying, but phantomcircuit gave an alternative argument that the reduction in spammy transactions relative to non-spammy ones may be resulting in lower cache hitrates.
08:50 < morcos> eyeballing the numbers i'm not seeing a noticeable slowdown since march
08:51 <@wumpus> which probably means it's a combination of both
08:52 <@wumpus> in any case the increased default dbcache should alleviate either problem, at least for a while
08:53 < morcos> ok, yeah my numbers are with a big dbcache (2G)
08:53 <@wumpus> there may be a better way to structure the utxo set on disk - but having looked into other databases it seems that leveldb has the best performance for our use, at least during sync
08:54 <@wumpus> if you set a huge dbcache you won't notice much, as sipa says it's probably why the surprise that indexing was so slow with default dbcache
08:54 < sipa> for 0.14 we should prioritize working on chainstate backups
08:55 < sipa> (and vigorously fight the potential push for servers with 'helpful' backups to download...)
08:58 < bsm1175321> I need to fund transactions and be sure that their txid is not malleable.  I was hoping 'fundrawtransaction' had an option to take only segwit inputs, but it seems it does not.  What's the best way to achieve this?  (I'm willing to write such an option if there isn't a better way) @sipa?
08:59 < sipa> bsm1175321: patches welcome :)
08:59 < sipa> seems like a very useful feature
08:59 < bsm1175321> Ok so the answer is that this isn't possible currently?
08:59 < sipa> for now, you'll need listunspent
08:59 < bsm1175321> Cool, I'll see what I can do.
09:00 < sipa> i believe i've heard someone else ask for the same feature
09:09 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev
09:09 < bsm1175321> One complication with such an option is that it's possible for your wallet to have enough funds to fund the transaction, but not enough segregated witness inputs.  A solution would be to create an intermediate transaction whose outputs have segregated witness.  Thoughts?
09:09 < sipa> fundrawtransaction shouldn't do such thing
09:09 < sipa> imho
09:10 < sipa> but it can fail with a nice error code, instructing you to clean up your wallet
09:10 < bsm1175321> You'd rather it fail in such a case?  That's fine with me too...
09:10 < sipa> yes, insufficient funds :)
09:10 < sipa> (or perhaps you should use separate wallets if dealing with that situation is hard)
09:10 < bsm1175321> What would you expect a user to do in such a case?  Is there a procedure to "clean up your wallet"?  That people will know?
09:11 < sipa> it's no different from the situation where you have a wallet with some watchonly coins, and the non-watchonly together are not enough to pay
09:12 < bsm1175321> True, sort of, except you have the funds.  I'll make a descriptive error message.
09:15 < sipa> i think the real solution is that if you have a need for segwit-only inputs, you run with a segwit-only wallet
09:17 < bsm1175321> In effect that's what I'll do.  But have to start from a non-segwit wallet.  Or is there a spent-to-myself-segwit-out wallet RPC call?
09:18 < sipa> you can just use sendtoaddress to send from the old wallet to the new?
09:19 < sipa> also, wallet integration of segwit is really preliminary at this point
09:19 < sipa> for example, change outputs won't be segwit
09:19 < bsm1175321> ooohhh...hmmm...
09:20 < sipa> (which they should be when it becomes ready for production, but that's not done yet)
09:21 < bsm1175321> Well if you 'fundrawtransaction ... segwitOnly' I can make it use a segwit change address.
09:21 < bsm1175321> Or you can add an explicit change address
09:22 < sipa> true
09:22 < bsm1175321> If no objections, I'll make the 'segwitOnly' option take only segwit inputs, and generate segwit change.
09:33 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
09:35 < instagibbs> what is/is there a way to link to an external secp library during compilation?
09:41 < bsm1175321> sipa: do you have a "wallet wishlist" for segwit?  I'm seeing that a lot of infrastructure is missing here.
09:43 < sipa> bsm1175321: thinks like default witness addresses
09:43 < sipa> post softfork
09:53 < bsm1175321> I'm confused regarding addresses.
09:54 -!- jtimon [~quassel@LAubervilliers-656-1-271-230.w193-248.abo.wanadoo.fr] has joined #bitcoin-core-dev
09:54 < bsm1175321> What is the output of 'addwitnessaddress'?  (what kind of address is that?)
09:55 -!- jtimon [~quassel@LAubervilliers-656-1-271-230.w193-248.abo.wanadoo.fr] has quit [Remote host closed the connection]
09:57 < sipa> p2sh
09:58 < bsm1175321> Ok, duh.  (hadn't made a testnet p2sh before, was confused by the 2)
10:04 < bsm1175321> Ok I think I understand.  A wallet bitcoin.conf really should have 3 possible values then: non-segwit, segwit, and segwit-in-p2sh.
10:04 < bsm1175321> *wallet setting
10:05 < bsm1175321> Do I understand correctly: The point of P2SH nesting is that non-upgraded wallets will check the destinations (though not the signatures)?
10:15 < jl2012> bsm1175321: I'd say the point of P2SH nesting is to allow non-upgraded wallets to send money to upgraded wallet
10:19 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 258 seconds]
10:20 < bsm1175321> Can't they always do that though?  Do you mean the other way around?
10:21 < arubi> it's made for upgraded nodes to be able to accept payment from unupgraded nodes to a segwit scriptpubkey, yes
10:23 < jl2012> for upgraded -> not upgraded, you just use P2PKH address (1xyz......)
10:25 < arubi> bsm1175321, I'm curious, what kind of malleability are you trying to avoid?  are you the one signing the transaction?  can't you use normal inputs too and just sign it properly?
10:26 < bsm1175321> arubi: I'm going to be handing out txids, including txids that may be in the mempool, not mined yet.
10:27 < arubi> signed by you and you only?  you could still "maul"(?) if you wanted to, even with segwit
10:28 < jl2012> arubi: without BIP62, that's not reliable
10:28 < bsm1175321> The non-upgraded nodes that receive funds from an upgraded node doesn't verify signatures though.  (correct?)
10:29 < arubi> jl2012, talking about the anyonecanpay|single sighash, sign once, permute how many you'd like
10:30 < arubi> bsm1175321, the script that they see doesn't talk about checksigs, so they don't check any signatures
10:30 < jl2012> arubi: there are still other forms of third party malleability, e.g. non-canonical push
10:30 < bsm1175321> arubi: signed by me and me only.
10:31 < arubi> jl2012, sure, using standard scripts is a must here if you don't want 3rd parties doing things, but as a single signer, you could still do it to your own txs, even with segwit
10:33 < jl2012> well, if you want to guarantee non-malleability to only yourself, segwit is the only way
10:33 < bsm1175321> jl2012: can you elaborate?  Malleation of the script (which is part of the witness data) doesn't change the txid, and I don't think I care about that.
10:33 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
10:34 < arubi> jl2012, true.  I am asking because bsm1175321 explains that he will hand out txids (not the transactions themselves, as I understand)
10:34 -!- pedrobranco [~pedrobran@79.242.108.93.rev.vodafone.pt] has quit [Remote host closed the connection]
10:34 < arubi> so as a single signer, he could still cause malleability, even if he manages to convince the other party all inputs are segwit
10:34 < arubi> *and standard scripts, or even better, even only with p2wpkh
10:34 < jl2012> arubi: that's double-spending
10:34 -!- pedrobranco [~pedrobran@79.242.108.93.rev.vodafone.pt] has joined #bitcoin-core-dev
10:35 < jl2012> not malleability
10:35 < arubi> how so?  payments are still paid, but it's the order of the inputs outputs that's changed
10:35 < arubi> you're using the same inputs
10:36 < bsm1175321> arubi: Are you saying I could change the order of outputs, keeping the same segwit txid?
10:36 < arubi> no, you're changing it
10:37 < bsm1175321> Ok then someone looking for one txid I handed them won't see it at all, and there's no point in me giving the wrong txid to someone.  (A txid which never get mined because I malleated it is useless)
10:38 < arubi> right
10:39 < arubi> they should be checking their addresses for incoming transactions, and that's it.  txid is something that's only relevant when spending
10:39 < arubi> *should only be, I guess
10:39 < bsm1175321> However if they act on a txid in the mempool, and then I broadcast a malleated version which gets mined.  That's your usual double spend.
10:39 -!- pedrobranco [~pedrobran@79.242.108.93.rev.vodafone.pt] has quit [Ping timeout: 252 seconds]
10:40 < arubi> how is it a double spend?  there is no spending from the malleated version
10:40 < pigeons> well if you only malleate the signature its not a double spend
10:40 < bsm1175321> Let's say I malleate the output order.  This is a weird thing to do.  I'll have to think on it...
10:41 < arubi> same can be done for inputs
10:42 < pigeons> if you have someone taking action on the txid of the unconfirmed transaction they will be affected, otherwise, not right
10:43 < arubi> a spender still has to reference a txid as input, so even if the outputs order isn't changed, the input is invalid in case of malleability
10:43 < arubi> oh you mean if the bad one wasn't mined, right
10:48 < bsm1175321> In this case, I'm proving control of coins.  Malleating my own txn is shooting myself in the foot.
10:55 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev
10:58 < arubi> I'm guessing, you're proving control by announcing the txid before the transaction is seen on the network?
11:08 < bsm1175321> Not *before* but it could be simultaneous.  And I'm wondering what the consequences are of fiddling around...
11:09 < bsm1175321> But If I claim to prove something with a txid, and you look for it on the network, I'm only hurting myself by messing with it.  I'm having trouble thinking of any negative consequences here.
11:11 < arubi> why not use signed messages to prove ownership of addresses?  as long you commit to an address, which is kinda like commiting to specific coins
11:12 -!- kadoban [~mud@unaffiliated/kadoban] has quit [Ping timeout: 250 seconds]
11:17 < arubi> hm.  I guess if it's a p2sh that's not so simple, because you'd have to disclose the redeemscript
11:18 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has joined #bitcoin-core-dev
11:23 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
11:26 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has quit [Quit: brb]
11:27 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has joined #bitcoin-core-dev
11:45 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Remote host closed the connection]
11:45 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
11:45 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Remote host closed the connection]
11:45 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
11:49 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Remote host closed the connection]
11:50 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
11:50 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Remote host closed the connection]
11:50 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
11:51 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Remote host closed the connection]
11:56 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
12:03 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Remote host closed the connection]
12:03 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
12:07 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has joined #bitcoin-core-dev
12:08 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Ping timeout: 240 seconds]
12:23 -!- TheFactory7 [uid164731@gateway/web/irccloud.com/x-jxaukbigjfvzpkje] has quit [Quit: Connection closed for inactivity]
12:31 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
12:31 -!- zooko [~user@2601:281:8000:8387:5157:31b2:4a62:d545] has joined #bitcoin-core-dev
12:33 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
12:34 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Quit: laurentmt]
12:36 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev
12:39 -!- Lysander1 [~Lysanders@unaffiliated/lysanders] has joined #bitcoin-core-dev
12:41 -!- CyrusV [~cyrus@unaffiliated/cyrusv] has quit [Ping timeout: 252 seconds]
12:41 -!- Cory [~C@unaffiliated/cory] has quit [Ping timeout: 252 seconds]
12:41 -!- Expanse [sid146237@gateway/web/irccloud.com/x-wcuzfvrdoztuukpy] has quit [Ping timeout: 252 seconds]
12:42 -!- jannes [~jannes@178.132.211.90] has quit [Ping timeout: 252 seconds]
12:42 -!- Lysanders [~Lysanders@unaffiliated/lysanders] has quit [Ping timeout: 252 seconds]
12:42 -!- LeMiner [LeMiner@unaffiliated/leminer] has quit [Ping timeout: 252 seconds]
12:42 -!- OxADADA [~OxADADA@alumni-linux.ccs.neu.edu] has quit [Ping timeout: 252 seconds]
12:42 -!- TD-Linux [~Thomas@about/essy/indecisive/TD-Linux] has quit [Ping timeout: 252 seconds]
12:42 -!- nanotube [~nanotube@unaffiliated/nanotube] has quit [Ping timeout: 252 seconds]
12:42 -!- crescendo [~mozart@unaffiliated/crescendo] has quit [Ping timeout: 252 seconds]
12:42 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has quit [Ping timeout: 252 seconds]
12:42 -!- kinlo [peter@unaffiliated/kinlo] has quit [Ping timeout: 252 seconds]
12:42 -!- hexis [~hexis@208.87.122.37] has quit [Ping timeout: 252 seconds]
12:42 -!- petertodd [~pete@ec2-52-5-185-120.compute-1.amazonaws.com] has quit [Ping timeout: 252 seconds]
12:42 -!- windsok [~windsok@45.63.59.8] has quit [Ping timeout: 252 seconds]
12:42 -!- windsok_ [~windsok@45.63.59.8] has joined #bitcoin-core-dev
12:43 -!- Pasha [~C@unaffiliated/cory] has joined #bitcoin-core-dev
12:43 -!- crescendo [~mozart@unaffiliated/crescendo] has joined #bitcoin-core-dev
12:44 -!- OxADADA [~OxADADA@alumni-linux.ccs.neu.edu] has joined #bitcoin-core-dev
12:44 -!- kinlo [peter@unaffiliated/kinlo] has joined #bitcoin-core-dev
12:44 -!- petertodd [~pete@ec2-52-5-185-120.compute-1.amazonaws.com] has joined #bitcoin-core-dev
12:44 -!- adiabat [~tx@159.203.193.74] has joined #bitcoin-core-dev
12:45 -!- Expanse [sid146237@gateway/web/irccloud.com/x-eucjrihqwfzgmahw] has joined #bitcoin-core-dev
12:45 < adiabat> hey, I'm spamming testnet and have some unexpected behavior
12:45 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Quit: laurentmt]
12:46 < adiabat> I think I get what "size", "cost", "strippedsize", "Vsize" all mean
12:46 < adiabat> "cost" is basically "Vsize" * 4
12:47 < adiabat> so a block "cost" must be < 4M
12:47 -!- cryptapus is now known as cryptapus_afk
12:47 < adiabat> (<=, whatever)
12:47 -!- TD-Linux [~Thomas@about/essy/indecisive/TD-Linux] has joined #bitcoin-core-dev
12:48 < adiabat> so in .conf, blockmaxsize should set the cost limit of created blocks
12:48 < adiabat> but it seems to be targeting "size" instead
12:48 -!- CyrusV [~cyrus@unaffiliated/cyrusv] has joined #bitcoin-core-dev
12:49 < bsm1175321> arubi: I want to be able to commit to coins regardless of the address type, and as you say, I'd need the redeemScript.
12:50 -!- Pasha is now known as Cory
12:52 -!- kvnn [~kvnn@cpe-108-185-240-254.socal.res.rr.com] has joined #bitcoin-core-dev
12:53 -!- nanotube [~nanotube@unaffiliated/nanotube] has joined #bitcoin-core-dev
12:54 < kvnn> Hello everyone. I'm offering a 1BTC bounty for 3&4 here: https://github.com/drivechain-project/docs . Please let me know if you are interested. (if this is not okay in this channel I apologize & will discontinue posting about it)
12:55 -!- jannes [~jannes@178.132.211.90] has joined #bitcoin-core-dev
12:55 -!- hexis [~hexis@208.87.122.37] has joined #bitcoin-core-dev
12:56 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has joined #bitcoin-core-dev
12:59 -!- sdaftuar_ [uid164573@gateway/web/irccloud.com/x-lujqtzaxchysvunn] has joined #bitcoin-core-dev
13:02 < sdaftuar_> adiabat: -blockmaxsize continues to refer to total serialized bytes of a block, counting witness and non-witness parts the same. -blockmaxcost is the option that specifies the limit you probably want here.
13:04 < adiabat> hmmm ok
13:04 < adiabat> so there's both
13:09 < sdaftuar_> Yeah.  The hope is that -blockmaxsize is deprecated in the future.
13:10 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has quit [Quit: back in a lot]
13:10 < sdaftuar_> (My preference was to change the semantics of -blockmaxsize to refer to segwit's virtual size, but that's not what we ended up with)
13:11 < luke-jr> well, depends what you want to limit
13:11 < roasbeef> segwit miners on testnet seem to be limiting to ~750k cost
13:11 < luke-jr> sdaftuar_: then miners would need to set blockmaxsize to 250k to avoid making blocks >1M
13:11 < gmaxwell> 06:43 < MarcoFalke> gmaxwell: I don't think lack of progress in improving of coinselection is due to the test hardcoding the behavior.
13:12 < luke-jr> the goal of blockmaxsize is to avoid blocks larger than the size. cost/vsize doesn't matter here
13:12 < gmaxwell> I can tell you that personally I've written improvements that I haven't PRed because they required throwing out all the existing tests.
13:12 < sdaftuar_> My preference was to introduce a new option that would control serialized bytes
13:12 < luke-jr> sdaftuar_: why a new oppppption when we have one for it already?
13:13 < sdaftuar_> Because the mining code doesn't optimize for it
13:13 < sdaftuar_> So it's misleading to suggest it's supported
13:14 < luke-jr> it does what it's supposed to do.
13:14 < gmaxwell> I don't think it makes sense to even have a seralized size setting, we don't have a setting to limit the size of the sighashed bytes, or a setting to limit the size of the block's CTransaction encoding.  We don't have a limit to control the size of the compactblock form, or a limit to control the size of a zlib compressed block.
13:14 < gmaxwell> But I don't bother complaining because luke was going pyric over this before, and it shouldn't hurt that much having it either.
13:14 < luke-jr> gmaxwell: at this time, serialised size is a critical factor in practice
13:15 < sdaftuar_> gmaxwell: +1
13:15 < gmaxwell> luke-jr: critical for what? it doesn't reflect block propagation time especially well. 
13:15 < luke-jr> on the p2p network it sure does..
13:16 < gmaxwell> validation time is more closely related to the number of utxo operations and how well relayed the txn in hte block were.
13:16 < gmaxwell> luke-jr: we have BIP152 now.
13:16 < luke-jr> not deployed, unfortunately.
13:17 < luke-jr> to be clear, I'm all for removing blockmaxsize once it doesn't matter and isn't used.
13:17 < luke-jr> hence why I don't think optimising block creation for it is important
13:17 < gmaxwell> (but even without it, seralized size is still not the ideal predictor of propagation time.)
13:18 < luke-jr> I just worry that we'll end up with >1 MB blocks before the network can handle it sanely
13:18 < gmaxwell> K. indeed 0.12 doesn't have BIP152. Perhaps that does suggest that it should default to four million in 0.13?
13:18 < gmaxwell> luke-jr: thats inevitable...
13:18 < gmaxwell> see also the utxo comments by wumpus last night.
13:24 -!- Cats2 is now known as ClockCat
13:30 -!- zooko [~user@2601:281:8000:8387:5157:31b2:4a62:d545] has quit [Ping timeout: 250 seconds]
13:47 < sturles> Am I correct that the 0.13 HD wallet implementation will only support new wallets, and only support hardened keys?
13:47 < gmaxwell> sturles: correct.
13:48 < sturles> Will it support importing private keys from an old style wallet?
13:48 < gmaxwell> I believe it does, I've not actually tested that!
13:53 < sturles> A HD wallet won't work with older versions, right?  Should use the opportunity to switch DB for the wallet to something > libdb 4.8?
13:54 < gmaxwell> Do you want software that never makes progress? Thats how you get software that never makes progress.
13:54 < gmaxwell> It can't use a later libdb or it will make all non-HD wallets also unreadable by other versions.
13:55 < helo> i believe a hd wallet should work with older versions
13:56 < gmaxwell> helo: hows that?
13:56 < sturles> Yes, I know.  I guess it is impossible to use different versions depending on wallet type, e.g. via dlopen?
13:57 < gmaxwell> sturles: in theory possible but that would be a lot of work without any benefit.
13:57 < helo> it will work to a limited extent... hd keys already added will be visible
13:57 < helo> untested, ofc
13:57 < gmaxwell> if so, thats a bug. it should get rejected due to having a too new version.
13:58 < Lightsword> can we just migrate to sqlite?
13:58 < gmaxwell> if it isn't then you could put yourself in a weird state.
13:58 < gmaxwell> I will have to start stabbing people.
13:59 < helo> nah, that's not necessary. the version check probably functions as intended :)
13:59 < gmaxwell> hah
14:01 < sturles> I'd say it is benificial to switch to a newer, supported version of libdb
14:02 < gmaxwell> the latest versions have incompatible licenses.
14:04 < sturles> Oh.
14:05 < gmaxwell> there is newer than 4.8 that is okay, but the latest stuff has a much stronger copyleft than we'd normally use.
14:07 < gmaxwell> really the 'database' isn't actually used for much--- the wallet is largely entirely in memory, and the database is only really used for persistance.
14:08 -!- murch [~murch@p4FE3A233.dip0.t-ipconnect.de] has quit [Ping timeout: 264 seconds]
14:09 -!- murch [~murch@p4FE3949A.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
14:14 < luke-jr> eh, -walletupgrade won't work? O.o
14:30 -!- afk11 [~afk11@unaffiliated/afk11] has quit [Ping timeout: 240 seconds]
14:36 < Lightsword> would working on migrating the wallet to sqlite be something worthwhile at this point?
14:38 -!- afk11 [~afk11@109.255.154.81] has joined #bitcoin-core-dev
14:38 -!- afk11 [~afk11@109.255.154.81] has quit [Changing host]
14:38 -!- afk11 [~afk11@unaffiliated/afk11] has joined #bitcoin-core-dev
14:39 < CubicEarth> Block validation is, theoretically, very amenable to parallelization, correct?  Is the main serial component the depth of the merkel tree?
14:39 -!- murch [~murch@p4FE3949A.dip0.t-ipconnect.de] has quit [Quit: Leaving.]
14:42 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
14:43 < gmaxwell> signature validation is, and bitcoin core runs that in paralle... but normally almost all of the signatures in a block are already validated before it shows up.
14:44 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 250 seconds]
14:44 < gmaxwell> parallel*
14:44 < gmaxwell> the general database handling ends up taking much of the time, which isn't really paralizable. 
14:48 -!- mkarrer_ [~mkarrer@48.red-83-47-111.dynamicip.rima-tde.net] has quit []
14:51 -!- sdaftuar1 [~sdaftuar@unaffiliated/sdaftuar] has joined #bitcoin-core-dev
14:51 -!- ybit_ [~ybit@unaffiliated/ybit] has joined #bitcoin-core-dev
14:51 -!- OxADADA_ [~OxADADA@alumni-linux.ccs.neu.edu] has joined #bitcoin-core-dev
14:51 -!- trippysalmon [~trippy@cyberdynesys.org] has joined #bitcoin-core-dev
14:52 -!- mkarrer [~mkarrer@48.red-83-47-111.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
14:53 -!- tadasv [ttttt@gateway/shell/panicbnc/x-unhqyribepejyjop] has quit [Excess Flood]
14:53 -!- [b__b] [~b__b]@ec2-54-85-45-223.compute-1.amazonaws.com] has quit [Remote host closed the connection]
14:53 -!- sdaftuar [~sdaftuar@unaffiliated/sdaftuar] has quit [Remote host closed the connection]
14:53 -!- OxADADA [~OxADADA@alumni-linux.ccs.neu.edu] has quit [Remote host closed the connection]
14:53 -!- trippysa1mon [~trippy@cyberdynesys.org] has quit [Remote host closed the connection]
14:53 -!- ybit [~ybit@unaffiliated/ybit] has quit [Remote host closed the connection]
14:53 -!- sdaftuar_ is now known as sdaftuar
14:53 -!- tadasv [ttttt@gateway/shell/panicbnc/x-aowitmmojkiaruoe] has joined #bitcoin-core-dev
14:54 -!- [b__b] [~b__b]@ec2-54-85-45-223.compute-1.amazonaws.com] has joined #bitcoin-core-dev
14:55 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Read error: Connection reset by peer]
15:01 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-dev
15:05 < CubicEarth> gmaxwell: so thinking about the initial chain sync and validation, it could (with lots and lots of careful work) be made to harness GPU's, and spread the signature validation across *many* cores?  I'm not familiar with the nature of the general database handling you are referring to, but for the initial sync is that mainly keeping track of the utxo's as it increments tough the blocks?
15:05 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 258 seconds]
15:07 < gmaxwell> I've seen no reason to believe that a gpu would help at all, GPU cores aren't particularly good at the work of validating. (64 bit arithmetic helps a lot!)
15:08 < gmaxwell> and nodes don't even verify the signatures far back in the chain.
15:08 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
15:08 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-dev
15:09 < gmaxwell> but sure more work could be done to extract more parallelism out of it.
15:11 < gmaxwell> but even with signature validation turned off completely, with default settings a reindex takes almost 9 hours currently.
15:13 < gmaxwell> increasing the db cache to a really huge size (such that the sync runs entirely out of ram) lets the whole sync _with_ signature checking run in about 3.5 hours. 
15:14 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has joined #bitcoin-core-dev
15:14 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev
15:16 < CubicEarth> Is the default 100MB?  I never knew that!!  I'll be curious too see what kind of speedup I can achieve on an i5 - dual core.  Are we talking 100GB of ram here, or would setting it to 4 or 8 GB make a substantial difference?
15:17 < sipa> setting it to 2 GB is more than sufficient
15:17 < sipa> higher numbers don't matter much
15:18 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
15:19 < CubicEarth> Is it a rolling-window thing?
15:19 < sipa> no
15:19 < sipa> it's a cache
15:19 < sipa> when the cache is full, we write it to disk
15:19 < sipa> and start over
15:19 < sipa> very silly
15:22 < gmaxwell> I think you actually need more than 2GB now to get full performance, but 8GB is more than enough.
15:22 < midnightmagic> maybe an iovec scatter write might help some systems write faster
15:24 < midnightmagic> or is it just a plain sequential write thing and the rest is leveldb
15:24 < CubicEarth> Well it's easy enough to set it higher.  I just never knew.  A default of 100MB can make sense for older systems, but if there was place for 'performance tips', that would be a good place to let people know.  Maybe everyone already (node operators) knows except for me.
15:25 < gmaxwell> I think a lot of people don't realize what a big difference it makes.
15:25 < gmaxwell> looks like we'll increase the default to 300MB.
15:26 < gmaxwell> wumpus: on the leveldb stuff, without txindex, I'd suggest making that 4mb instead of 2. 4MB was something like 1% faster for me when testing leveldb in isolation.
15:27 < gmaxwell> maybe 2 is a win at the 300mb limit, but with it bumped 4mb would be... and probably is more conservative overall.
15:31 -!- sdaftuar [uid164573@gateway/web/irccloud.com/x-lujqtzaxchysvunn] has quit [Quit: Connection closed for inactivity]
15:35 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
15:36 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)]
15:39 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Ping timeout: 246 seconds]
15:39 < CubicEarth> I did a fresh sync a week ago, 2013-era i5 dual core laptop.  Ubuntu, 4GB of ram, spinning HD.  75Mbps cable internet.  It took about 20 hours to finish.  What was interesting though was how the last 20 weeks just crawled.  Seemed less than linear, even estimating about the effects of fuller blocks.
15:42 < sipa> yeah, try dbcache 4000 or so
15:43 < CubicEarth> after I order more ram :)
15:47 < CubicEarth> gmaxwell: earlier you wrote "and nodes don't even verify the signatures far back in the chain.".  That's due to checkpoints, right?  They don't seem like a good thing to rely on.  I understand their practicality, but yeah.
15:49 < CubicEarth> I'm just trying to understand the theoretical performance limits... I'm not crusading against checkpoints
15:52 < gmaxwell> well you'd be welcome here to do that, I want them gone and have for years.
15:53 < gmaxwell> But even with them gone, one can avoid skipping signature validation during the initial sync for blocks burried by thousands of additional blocks with negligible risk-- consider, if miners went rogue enough to reorg thousands of blocks the system is already screwed... and if its only during initial sync they'd only trick new nodes in any case.
15:54 < gmaxwell> if it makes a difference between a user running a node or not-- the best choice is clear.
15:54 < gmaxwell> Meanwhile bitcoinxt and classic have disabled signature checking for any block where the miner supplied timestamp is a day back or more... meaning they can be fooled without a reorg at all. And no one seems to care.
15:56 < CubicEarth> Arn't miners only afforded the option to have their timestamp be off by more than 2-hours?
15:57 < CubicEarth> (as an aside)
15:57 < gmaxwell> no, not in the past direction. In the future direction yes.
15:58 < gmaxwell> The past direction is limited by the median of the last 11 blocks-- but that median can be arbritarily far back.
15:59 < CubicEarth> So practically speaking, the way to detect the issue is when the client said it was 'synced' and you noticed that is was reporting yesterdays date?
15:59 < midnightmagic> ..  did they actually commit that change? the >24hr non-t-verify change?
16:01 < gmaxwell> CubicEarth: no, because miners can simply add a correctly timed block immediately after.
16:02 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has quit [Ping timeout: 276 seconds]
16:02 < CubicEarth> That would appear as if the hashrate had fallen precipitously... not blocks for 24hr!
16:03 < gmaxwell> midnightmagic: XT did and released, it's merged in classic, but I don't know if it made it into a release yet.
16:03 < gmaxwell> These folks are extremely and dangerously incompetent.
16:04 < gmaxwell> CubicEarth: no, just a wonky timestamp, miners provide the timestamps and sometimes they're pretty wonky.
16:04 < gmaxwell> and while _syncing_ you have no idea that the timestamps were wonky. 
16:05 < gmaxwell> e.g. someone can produce blocks that are -24h, -23h, -22h .... normal.
16:05 < midnightmagic> so.. the peer preference thing means some of the alt-branch software will soon start disconnecting the only nodes that are immune to the attack, making themselves *more* vulnerable to it?
16:07 < gmaxwell> midnightmagic: I don't think that matters that much, but right now bitcoin "xt" is only not totally partitioned due to inbound connections from core to XT, but part of segwit is that segwit capable nodes will strongly prefer to connect out to only non-segwit nodes (because they must in order to get blocks once segwit is activated)-- the result will be making that partition complete.
16:08 < midnightmagic> i thought -classic was now disconnecting non-classic nodes
16:09 < gmaxwell> no, Xt. I don't think classic is doing that yet.
16:10 < slackircbridge1> <alp> let them lose money for people
16:10 < slackircbridge1> <alp> its the only way theyll learn
16:10 < gmaxwell> It's not that simple. See also: Ethereum and DAO.
16:11 < gmaxwell> which adversely impacted the bitcoin price even though many of us have been pointing out the related risks and distinguishing bitcoin on the basis of them for some time.
16:13 < slackircbridge1> <brg444> :O
16:13 < CubicEarth> gmaxwell: well I'm pretty sure the rise of ETH was hurting the bitcoin price to some extent.  I think this event was a decoupling, where Bitcoin shakes the monkey off its back.
16:13 < slackircbridge1> <brg444> did he read your post :O
16:16 -!- jannes [~jannes@178.132.211.90] has quit [Quit: Leaving]
16:24 < slackircbridge1> <alp> Short term pain, long term gain
16:27 < slackircbridge1> <moli> guys, we can read your posts on IRC lol
16:28 -!- mode/#bitcoin-core-dev [+o btcdrak] by ChanServ
16:32 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 276 seconds]
16:32 -!- mode/#bitcoin-core-dev [+scirg] by btcdrak
16:32 -!- mode/#bitcoin-core-dev [-c] by ChanServ
16:33 <@btcdrak> wait that's wrong. I'm trying to set slackircbridge1 +q
16:33 < gmaxwell> lol
16:34 < gmaxwell> undo the damage first.
16:36 -!- mode/#bitcoin-core-dev [+o gmaxwell] by btcdrak
16:37 -!- mode/#bitcoin-core-dev [-sirg] by gmaxwell
16:38 -!- mode/#bitcoin-core-dev [+q slackircbridge*!*@*] by gmaxwell
16:38 -!- mode/#bitcoin-core-dev [-o btcdrak] by btcdrak
16:42 -!- mode/#bitcoin-core-dev [-o gmaxwell] by gmaxwell
17:10 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
17:23 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
17:26 -!- TheFactory7 [uid164731@gateway/web/irccloud.com/x-fujyqdybnukpuobu] has joined #bitcoin-core-dev
17:28 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Ping timeout: 250 seconds]
18:16 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
18:17 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
18:40 -!- LeMiner [LeMiner@unaffiliated/leminer] has joined #bitcoin-core-dev
18:41 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
18:42 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
18:50 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 258 seconds]
18:54 -!- justanotheruser [~Justan@unaffiliated/justanotheruser] has quit [Quit: Lost terminal]
18:54 -!- justanotheruser [~Justan@unaffiliated/justanotheruser] has joined #bitcoin-core-dev
18:58 < adiabat> Hey I have a humble suggestion
18:58 < adiabat> doesn't really change functionality
18:58 < adiabat> right now in the getrawtransaction and getblock rpc calls
18:58 < adiabat> there's size and Vsize for a tx
18:58 < adiabat> and strippedsize, size, and cost for a block
18:59 < adiabat> as far as I can tell, cost is just Vsize * 4
18:59 < gmaxwell> nope, you're not looking at segwit transactions.
18:59 < adiabat> ok wait, what's the relationship of cost and Vsize
19:00 < adiabat> I thought Vsize was normalized to 1M;  (Wsize + 3*strippedsize) / 4
19:03 < gmaxwell> It's (cost + 3) / 4; technically.  Cost is size + 3*strippedsize.
19:04 < adiabat> ok but so, Vsize is cost /4, round up.
19:04 < gmaxwell> yes.
19:05 < adiabat> could you say a tx has a cost?
19:05 < adiabat> seems like you could...
19:05 < gmaxwell> Yes I just did?
19:05 < adiabat> right, so I'm just suggesting, for rpc calls, return the cost of the tx
19:05 < gmaxwell> (there is a problem with using the word cost, annoyingly, it seems like it gets confused with fee)
19:06 < adiabat> yeahhhh I know
19:06 < gmaxwell> when it's just an abstract unit.
19:06 < adiabat> It's just that right now it's a bit confusing because blocks have cost and txs have Vsize
19:06 -!- bsm1175321 [~mcelrath@38.121.165.30] has quit [Read error: Connection reset by peer]
19:06 < adiabat> seems like you could say that txs also have cost with no real loss of information
19:07 < gmaxwell> they do, the vsize is strictly a lossy unit.
19:07 -!- justanotheruser [~Justan@unaffiliated/justanotheruser] has quit [Read error: Connection reset by peer]
19:08 -!- justanotheruser [~Justan@unaffiliated/justanotheruser] has joined #bitcoin-core-dev
19:08 < gmaxwell> I agree they should be consistent. And really, they should be the cost unit..
19:08 < gmaxwell> I wish we had a better word than cost.
19:08 < adiabat> OK cool that's what I'm saying
19:08 < adiabat> with txs now, you see:
19:08 < adiabat>   "size": 38919,
19:08 < adiabat>   "Vsize": 26433,
19:09 < adiabat> but you could return "size": 38919, "cost": 105732
19:09 < adiabat> that way it's more consistent and one less name to keep track of for users
19:11 -!- davec_ [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Ping timeout: 250 seconds]
19:18 -!- cryptapus_afk is now known as cryptapus
19:20 -!- davec_ [~davec@cpe-24-243-251-52.hot.res.rr.com] has joined #bitcoin-core-dev
19:44 -!- PaulCape_ [~PaulCapes@204.28.124.82] has joined #bitcoin-core-dev
19:48 -!- PaulCapestany [~PaulCapes@204.28.124.82] has quit [Ping timeout: 276 seconds]
19:49 < CubicEarth> gmaxwell:  Thinking about your 'carving a block of marble' analogy for forks, and thinking about the nature of Bitcoin, I tend to think that anything can be accomplished with a 'soft-fork', such as effectively raising the coin limit, etc.  I see it like a nested-universe scenario, but in the marble analogy... take the statue of David.  Say we find ourselves disappointed that Michelangelo didn't give David 
19:49 < CubicEarth> two heads.  We could basically carve a miniature replica of David out of the right butt-cheek of the full size version, with the 'replica' having two heads.  In real life there would be problems with absolute scale, but I don't see a parallel in code.
19:50 < CubicEarth> Perhaps I am taking the analogy too far.
19:51 < gmaxwell> You can try, but it's not invisible or non-consentual. 
19:51 < gmaxwell> Imagine that you try to make more coins.. well no existing wallet will reconize or accept those coins.
19:51 < gmaxwell> you pay me some 0 value output that the softfork reconizes as the extra coins. fine.. but my wallet will see it as zero.
19:52 < CubicEarth> That is true
19:52 < CubicEarth> Thankfully
19:52 < gmaxwell> so the most you could do is try to also DOS attack me to force me into going along with that change and accepting those coins, but that reduces to a power miners always have (DOS attacking users) which has a tidy solution: change the POW to fire the service provider (miners) that are screwing over the users and not faithfully including ordinary transactions.
19:55 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-lmtosqfecaezoaiz] has quit [Quit: Connection closed for inactivity]
20:03 -!- TheFactory7 [uid164731@gateway/web/irccloud.com/x-fujyqdybnukpuobu] has quit [Quit: Connection closed for inactivity]
20:04 -!- cryptapus [~cryptapus@unaffiliated/cryptapus] has quit [Quit: conversation terminated!]
20:10 -!- cryptapus [~cryptapus@unaffiliated/cryptapus] has joined #bitcoin-core-dev
21:40 -!- adiabat [~tx@159.203.193.74] has quit [Quit: Leaving.]
21:44 -!- kvnn [~kvnn@cpe-108-185-240-254.socal.res.rr.com] has quit [Remote host closed the connection]
21:45 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
21:46 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
21:47 -!- netsin [~jiggalato@198.16.176.10] has joined #bitcoin-core-dev
21:47 -!- netsin [~jiggalato@198.16.176.10] has quit [Client Quit]
21:47 -!- jiggalator [~jiggalato@unaffiliated/jiggalator] has joined #bitcoin-core-dev
21:48 -!- jiggalator is now known as netsin
21:51 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has quit [Quit: Leaving.]
21:53 -!- netsin [~jiggalato@unaffiliated/jiggalator] has quit [Remote host closed the connection]
22:03 -!- CyrusV` [~cyrus@unaffiliated/cyrusv] has joined #bitcoin-core-dev
22:04 -!- Netsplit *.net <-> *.split quits: CubicEarth, tadasv, whphhg, jonasschnelli, michagogo, CyrusV
22:05 -!- jonasschnelli_ [~jonasschn@2a01:4f8:200:7025::2] has joined #bitcoin-core-dev
22:23 -!- whphhg [whphhg@gateway/vpn/mullvad/x-ztjnijvjzmzcjibz] has joined #bitcoin-core-dev
22:23 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has joined #bitcoin-core-dev
22:30 -!- cryptapus is now known as cryptapus_afk
22:31 -!- netsin [~jiggalato@unaffiliated/jiggalator] has joined #bitcoin-core-dev
22:35 -!- CubicEar_ [~cubiceart@2600:100f:b010:3580:94cb:4882:a4af:45dd] has joined #bitcoin-core-dev
22:39 -!- CubicEar_ [~cubiceart@2600:100f:b010:3580:94cb:4882:a4af:45dd] has quit [Read error: Connection reset by peer]
22:39 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has quit [Ping timeout: 264 seconds]
22:39 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has joined #bitcoin-core-dev
22:48 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has joined #bitcoin-core-dev
22:53 -!- pedrobranco [~pedrobran@167.225.61.94.rev.vodafone.pt] has quit [Ping timeout: 250 seconds]
22:55 -!- netsin [~jiggalato@unaffiliated/jiggalator] has quit [Remote host closed the connection]
23:03 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
23:04 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
23:24 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
23:25 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
23:33 -!- netsin [~jiggalato@unaffiliated/jiggalator] has joined #bitcoin-core-dev
23:34 -!- netsin [~jiggalato@unaffiliated/jiggalator] has quit [Remote host closed the connection]
23:41 -!- baldur [~baldur@pool-72-69-25-42.nycmny.fios.verizon.net] has quit [Ping timeout: 258 seconds]