--- Day changed Wed Nov 18 2015 00:10 < GitHub23> [bitcoin] laanwj opened pull request #7051: ui: Add "Copy raw transaction data" to transaction list context menu (master...2015_11_transaction_hex2) https://github.com/bitcoin/bitcoin/pull/7051 00:11 -!- jcorgan [~jcorgan@unaffiliated/jcorgan] has quit [Ping timeout: 240 seconds] 00:11 -!- jcorgan [~jcorgan@ec2-54-67-38-167.us-west-1.compute.amazonaws.com] has joined #bitcoin-core-dev 00:11 -!- jcorgan [~jcorgan@ec2-54-67-38-167.us-west-1.compute.amazonaws.com] has quit [Changing host] 00:11 -!- jcorgan [~jcorgan@unaffiliated/jcorgan] has joined #bitcoin-core-dev 00:20 < GitHub147> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/e8df8a5077df...f3ea48ad8b85 00:20 < GitHub147> bitcoin/master e855b01 Alex Morcos: Fix debug log message for block files 00:20 < GitHub147> bitcoin/master f3ea48a Gregory Maxwell: Merge pull request #7050... 00:20 < GitHub197> [bitcoin] gmaxwell closed pull request #7050: Fix debug log message for block files (master...nitfix) https://github.com/bitcoin/bitcoin/pull/7050 00:50 < GitHub3> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/f3ea48ad8b85...7f8e90da335e 00:50 < GitHub3> bitcoin/master 013a364 MarcoFalke: [contrib] Delete test-patches 00:50 < GitHub3> bitcoin/master 7f8e90d Wladimir J. van der Laan: Merge pull request #7030... 00:50 < GitHub125> [bitcoin] laanwj closed pull request #7030: [contrib] Delete test-patches (master...MarcoFalke-2015-contribC) https://github.com/bitcoin/bitcoin/pull/7030 00:53 -!- ParadoxSpiral [~ParadoxSp@p508B96EE.dip0.t-ipconnect.de] has joined #bitcoin-core-dev 00:58 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev 01:27 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 01:40 -!- cjcj [2e3b026a@gateway/web/freenode/ip.46.59.2.106] has joined #bitcoin-core-dev 01:45 -!- rubensayshi [~ruben@91.206.81.13] has joined #bitcoin-core-dev 02:08 -!- jtimon [~quassel@74.29.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev 02:26 -!- cjcj [2e3b026a@gateway/web/freenode/ip.46.59.2.106] has quit [Quit: Page closed] 02:31 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.] 03:15 < GitHub184> [bitcoin] MarcoFalke opened pull request #7052: [qa] python-bitcoinrpc is no longer a subtree (master...MarcoFalke-2015-qaSubtree) https://github.com/bitcoin/bitcoin/pull/7052 03:16 -!- MarcoFalke [8af6027e@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.126] has joined #bitcoin-core-dev 03:51 < GitHub111> [bitcoin] jtimon opened pull request #7053: Globals: Remove a bunch of Params() from main.cpp before 0.12 (master...globals-chainparams-main) https://github.com/bitcoin/bitcoin/pull/7053 03:54 < GitHub25> [bitcoin] jtimon closed pull request #5970: DEPENDENT: Globals: Avoid calling Params() (master...chainparams_cleanup) https://github.com/bitcoin/bitcoin/pull/5970 04:33 -!- guest234234 [~guest2342@171.4.112.141] has joined #bitcoin-core-dev 04:47 -!- rav3n_pl [~rav3n_pl@91.235.254.37] has quit [Ping timeout: 246 seconds] 05:21 -!- MarcoFalke [8af6027e@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.126] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 05:23 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has joined #bitcoin-core-dev 05:23 -!- sdaftuar [~sdaftuar@static-100-38-11-146.nycmny.fios.verizon.net] has joined #bitcoin-core-dev 05:28 < GitHub151> [bitcoin] jonasschnelli pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7f8e90da335e...03403d8c0f3b 05:28 < GitHub151> bitcoin/master 513686d MarcoFalke: [qt] Use maxTxFee instead of 10000000 05:28 < GitHub151> bitcoin/master 03403d8 Jonas Schnelli: Merge pull request #6951... 05:28 < GitHub190> [bitcoin] jonasschnelli closed pull request #6951: [qt] Use maxTxFee instead of 10000000 (master...MarcoFalke-2015-qtMaxFee) https://github.com/bitcoin/bitcoin/pull/6951 05:46 -!- zooko [~user@2601:281:8001:26aa:8874:a77d:6fe6:611e] has quit [Ping timeout: 246 seconds] 05:54 < dcousens> out of interest, why username:password vs just token for the RPC? 05:54 < dcousens> they're both sent in the clear right? 05:56 -!- Sanjay [~d@103.40.137.5] has joined #bitcoin-core-dev 06:01 < dcousens> ping gmaxwell: thoughts on https://github.com/bitcoin/bitcoin/pull/7044#issuecomment-157719965 06:01 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has joined #bitcoin-core-dev 06:02 -!- MarcoFalke [8af6027e@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.126] has joined #bitcoin-core-dev 06:06 < sipa> dcousens: if it was just a token, you wouldn't know which one to try 06:06 < sipa> dcousens: which is needed if you want strengthening 06:07 -!- MarcoFalke [8af6027e@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.126] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 06:15 < jonasschnelli> dcousens: The hashing improves the attack vectors a little bit (tiny, i agree). Isn't the hashing is a little step towards a http-auth-digest likes authentication? 06:20 -!- guest234234 [~guest2342@171.4.112.141] has quit [Ping timeout: 260 seconds] 06:22 < gmaxwell> dcousens: username is useful because you can log it. also it fits into the http auth scheme. 06:22 < gmaxwell> of course, you could set all your usernames to x and just use passwords. 06:22 < gmaxwell> oh yea, sipas point too. 06:30 < jgarzik> jonasschnelli, http digest auth has been brought up quite a few times... 06:33 < jonasschnelli> jgarzik: Yes. I once looked at a possible implementation and decided that it's not worth doing it. 06:34 < gmaxwell> It's also not compatible with a surprising amount of stuff. 06:34 < jgarzik> the question really comes down to SSL, IMO. We are sending plaintext passwords. Is that a problem on localhost? no. Is that a problem on WAN? Yes. Should you be sending unencrypted RPC over WAN? No, use stunnel. So now you're back to not needing digest. 06:35 < gmaxwell> right. 06:40 < jonasschnelli> agreed 06:40 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Quit: poof] 06:41 < wumpus> right: if you want secure RPC, tunnel it over something secure. Digest auth would be pretty pointless. Yes, you protect the auth, but your walletpassphrase will still go over the line unencrypted. 06:43 < wumpus> a challenge/response authentication system with the key storage/signer would be nice, as it'd also solve the 'walletpassphrase leaks into process memory' problem, but then again, you'd still be leaking actual commands over the line, which can (at the least) hurt privacy 06:43 < jonasschnelli> Right. #7044 (multiple RPC users) is interesting and it would allow to make permissions levels 06:44 < wumpus> so no, I think digest auth (or seemingly more secure) RPC auth is optimziing the wrong thing 06:44 < wumpus> but we want to get rid of the account stuff; what would be the use of multiple RPC accounts? 06:45 < wumpus> if you have multiple, completely separate wallets it could make sense I guess 06:46 < dcousens> gmaxwell: sure, but he could have been showing you a curl script with the rpcpassword in it 06:46 < jonasschnelli> companies that run services based on bitcoind could have multiple users, one that only accessed public data (like blocks / transaction), another user could have permissions to change things, like ban, etc. 06:46 < dcousens> the use case just isn't relevant imho, since, I doubt he minded lol 06:47 < gmaxwell> wumpus: not for seperate wallets, but access list management purposes; e.g. you with to withdraw one host, and add a new one. Otherwise you must disrupt everything tfor a flag-day change, and take an outage. 06:47 < wumpus> I don't want a complex, complete authentication and permissions system in bitcoin core 06:47 < wumpus> we're not buliding an OS here 06:47 < wumpus> it has to stop somewhere with importing all kind of ancillary concens 06:47 < dcousens> gmaxwell: I'm just not sure if its worth adding, since it'll give people impression that RPC is safer across a non-localized env? Weren't we discouraging that? 06:47 < jonasschnelli> Right. This could make things really complicate. 06:47 < wumpus> do you want access control? build an RPC proxy and have that check user permissions 06:48 < gmaxwell> Sure, but this is basic access management that any other network facing application provides (and basically any web api provides) 06:48 < wumpus> sigh... 06:48 < jonasschnelli> Yes. Thats a better approach. 06:48 -!- wumpus [~quassel@pdpc/supporter/professional/wumpus] has left #bitcoin-core-dev ["http://quassel-irc.org - Chat comfortably. Anywhere."] 06:48 < dcousens> (crap, just saw all the text above, notifications daemon died, reading now) 06:48 < dcousens> gmaxwell: isn't the point that this doesn't face the network? 06:48 < dcousens> All the docs have warnings to avoid that 06:49 < gmaxwell> It most certantly does-- not public networks, but even localhost is a network. Stunnel to it is network access. 06:49 < gmaxwell> If it's not facing things it cannot be used at all. 06:49 < dcousens> Sure, in which case, your on the machine, and, unless its a multi-user env, the hashing is pointless? 06:49 < gmaxwell> The secondary use, not enabled yet, is so that logging can also record which account did it, which is useful for debugging and forensics. 06:49 < dcousens> by multi-user env, I mean, the syystem itself, not bitcoind 06:50 < gmaxwell> The fact that the credentials are hased is a basic security practice that prevents leaking them when handling the credentials themselves; it's also a checklist requirement on many security standards (and not an unreasonable one) 06:51 < sipa> a single sha256 would suffice for that, though 06:52 < gmaxwell> It would. and if the credential is properly machine generated, then nothing more is actually useful. 06:53 < dcousens> gmaxwell: for logging, you could even just use hash(token), not untraceable 06:53 < dcousens> but even then, fine, user:pass still if you want make that easier 06:54 < gmaxwell> dcousens: doesn't really fit how people use things, the http auth already sends a user/password in any case. 06:54 -!- Greyboy [~Dingus@ec2-54-229-249-17.eu-west-1.compute.amazonaws.com] has joined #bitcoin-core-dev 06:54 -!- MarcoFalke [8af6027e@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.126] has joined #bitcoin-core-dev 06:54 < dcousens> gmaxwell: it just seems to me like the hashed password is implying a different security model to what has been advised up until now 06:55 < dcousens> and, aside from the auditing aspect, the username:password scheme is also misleading since its entirely in the clear 06:55 < dcousens> so, I guess, are we changing the security model? or keeping it the same 06:56 < gmaxwell> dcousens: almost 40 years of system security says otherwise. 06:56 < dcousens> gmaxwell: I have *no* issue with hashing, if we're protecting against local breach 06:56 -!- Greyboy [~Dingus@ec2-54-229-249-17.eu-west-1.compute.amazonaws.com] has quit [Changing host] 06:56 -!- Greyboy [~Dingus@unaffiliated/greyboy] has joined #bitcoin-core-dev 06:56 < dcousens> but, AFAIK, we haven't protected that to date 06:56 < gmaxwell> Seriously, people get _fired_ for keeping unhashed password databases. When a breach results from one it is an embarssment that results in litigation. 06:57 < GitHub96> [bitcoin] ptschip opened pull request #7055: MAX_PROTOCOL_MESSAGE_LENGTH description (master...maxmessage) https://github.com/bitcoin/bitcoin/pull/7055 06:58 < dcousens> gmaxwell: does that make sense? I'm asking if we're wanting to be more permissive in telling users "local breach isn't catastrophic" 06:58 -!- wumpus [~quassel@pdpc/supporter/professional/wumpus] has joined #bitcoin-core-dev 06:58 < dcousens> To date, that isn't the case 07:00 < Greyboy> I'm late to this convo, but like security. may i ask what passwords you are discussing? 07:01 * instagibbs reading backlog 07:03 < instagibbs> Greyboy, https://github.com/bitcoin/bitcoin/pull/7044 07:04 < Greyboy> Thank you. 07:04 < dcousens> instagibbs: good PR :), just noticed it changes the security model, so, just wanted that to be clear, because it'll likely mean we'll need to update docs which currently mention "dont ever expose the RPC", and, "dont use SSL", etc 07:04 < instagibbs> yeah I'm thinking through all this too 07:05 < dcousens> and everything being in the clear made that pretty obvious, whereas, now this seems half-half, in what was previously a blatantly unprotected env 07:10 < dcousens> wumpus: lol: https://en.bitcoin.it/wiki/Enabling_SSL_on_original_client_daemon 07:10 < Greyboy> instagibbs, i know i'm new and my opinion is relatively worthless, but i think it's a good idea. my question is what percent of people who run bitcoin-core actually need this solution? 07:11 < wumpus> dcousens: hah - the 0.12 release notes specify how to use stunnel as well as transparant proxying, would be more useful to add that 07:12 < sipa> Greyboy: or what percentage doesn't run it because it misses features like this? :) 07:12 < instagibbs> dcousens, I understand and agree with the concerns, but if we want some basic audit logging, I don't think we should avoid basic improvements 07:12 < dcousens> or more recently 07:12 < dcousens> https://bitcoin.stackexchange.com/questions/39117/why-is-json-rpc-over-ssl-strongly-discouraged 07:12 < instagibbs> keep the warnings, in general 07:12 < dcousens> though, thats a bit circular 07:12 < instagibbs> btw where are the warnings today 07:12 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has quit [Ping timeout: 250 seconds] 07:13 < dcousens> instagibbs: my main question doesn't stand in the way of doing logging 07:13 < dcousens> instagibbs: I'm just talking about the hashing, I'm ACK on everything else, even the hashing, just, clarifying that that is intended 07:13 < Greyboy> On a different note, I was wondering what bitcoin-core does that requires it to connect to dyndns? 07:13 < instagibbs> dcousens, would you feel better if it was a single hash :P 07:14 < gmaxwell> Well that would also drop any worry about where the pbkdf2 comes from. 07:14 < instagibbs> ^ 07:15 < instagibbs> warn people the same things we warned before (strong random keys only) 07:15 < gmaxwell> I don't consider the iterated hash actually important, it only potentially saves misusers from themselves slightly. 07:16 < dcousens> instagibbs: sure, I mean, the key-stretching seemed a bit unnecessary if we encourage what is already encouraged, strong random keys etc 07:17 < sipa> Greyboy: what? 07:17 < Greyboy> sipa, i was wondering why bitcoin-core tries to connect to dyndns and alerts my IDS 07:17 < instagibbs> I was going for "why not", but if it makes it more palatable, and strictly superior to today, and drops openssl dep.... 07:17 < sipa> Greyboy: what version? 07:18 < dcousens> anyway, its late here, hope I didn't stir up to much of a hornet nest haha. Just, recognizing this was a changed expectation. maybe its a better one, hard to tell 07:18 < Greyboy> I don't have it on this machine or i would check, but i've seen this behaviour in many versions including recent ones. 07:18 < instagibbs> dcousens, no this is really appreciated. My first Core PR getting tons of attention from smart folks is nice 07:21 < dcousens> instagibbs: jgarzik summed it up nicely 07:21 < dcousens> "the question really comes down to SSL, IMO. We are sending plaintext passwords. Is that a problem on localhost? no. Is that a problem on WAN? Yes. Should you be sending unencrypted RPC over WAN? No, use stunnel. So now you're back to not needing digest. 07:21 < dcousens> " 07:22 < instagibbs> I get the concern. Single hash at least gets the "peek over shoulder" security model. 07:22 < sipa> and sha256 we already have 07:22 < instagibbs> *the village rejoices* 07:22 < gmaxwell> dcousens: the digest thing is specifically referring to http digest auth, orthorgonal issue. 07:22 < dcousens> yeah just realised, was re-writing that 07:23 < dcousens> the other issue was, well, read up about 10 lines, night all 07:23 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has quit [Remote host closed the connection] 07:24 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has joined #bitcoin-core-dev 07:26 < instagibbs> dcousens, cheers 07:29 < sipa> if we want PBKDF2 in there, it's trivial to implement, but if we're going with just hashed passwords... even easier\ 07:34 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)] 07:35 < wumpus> right - I think we don't want something with a lot of key stretching, as it'd increase the latency for RPC calls 07:35 < wumpus> but hashing the passwords in the config file is a great idea, not sure why we've not been doing that yet 07:36 < sipa> we can even do salting by using HMAC-SHA256, but without stretching 07:36 < sipa> though i don't think that adds much 07:36 < sipa> except "certification" 07:36 < gmaxwell> if the key is randomly generated as it should be, the salting does nothing. 07:37 < gmaxwell> but yea, it might fail some checkbox. 07:37 < wumpus> salting is pretty standard for password checks, yes 07:37 < gmaxwell> I suggest the format be such that we could add different types in the future and if we find out its a problem we can just make a new password type. 07:37 < gmaxwell> or keep the salt, I don't care. :) 07:37 * wumpus wonders what Tor uses for torcontrolpassword 07:37 < instagibbs> I was just going to drop salt, but either way is fine 07:38 < instagibbs> we're assuming people aren't foot-gunning bad passwords already 07:38 < gmaxwell> only reason to keep it is certificational, which is an issue here. But it might be answerable with "the salt is _in_ the password" :) 07:39 < instagibbs> Excuse my noobishness, what is "certificational" about a salt 07:39 < sipa> instagibbs: rainbow tables! 07:39 < wumpus> tor apparently does salting, hashing 'bla' repeatedly gives 16:DD2D9F35664C69D7604862845D52D4238AD5E702E8C9DF771108459C00 16:EC737D2A880C18996066882BAE3BA73E32543DA0F74D308572288E38F0 16:35D59809F0AA181160F34D05D6504F89958E2A1D393A29959937C577A0 07:40 < gmaxwell> thats a +1 for salt. :P 07:40 < instagibbs> damnit 07:40 * instagibbs re-adds 07:40 < sipa> http://www.antitree.com/onion-porn-hashedcontrolpassword/ 07:40 < instagibbs> :P 07:40 < gmaxwell> instagibbs: the certificational part is that there are security checklists that ask if you're using salted hashed passwords. 07:40 < instagibbs> ohhhh like actualy certificates. Got it. 07:41 < wumpus> ah, thanks sipa, so 16 is the algorithm identifier, then follows the salt and password concatenated 07:41 < sipa> wumpus: nope, the 16 means it's encoded in hex :) 07:41 < sipa> seriously 07:41 < sipa> it's also the only thing supported 07:41 < wumpus> huh. ok 07:42 < sipa> but ehm... interoperability with tor would be nice imho 07:42 < wumpus> I'm confused with $3$ or /etc/shadow etc 07:42 < sipa> yeah 07:42 < gmaxwell> oh interesting actually thinking of using the same scheme as tor? is it complex to implement? 07:43 < sipa> 47 lines of python 07:43 < wumpus> we can just steal there code I guess :p although they may lean on OpenSSL as well 07:43 < Greyboy> *their 07:43 < sipa> it's just sha1 07:43 < sipa> iterated 07:45 < GitHub43> [bitcoin] ptschip closed pull request #7055: MAX_PROTOCOL_MESSAGE_LENGTH description (master...maxmessage) https://github.com/bitcoin/bitcoin/pull/7055 07:45 < wumpus> there's one difference though: tor only authenticates on opening the connection, bitcoin http authenticates for every request. So they can use more key stretching without creating command latency issues. 07:45 < gmaxwell> well we already have a sha1 dependency. 07:45 < instagibbs> hmm 07:46 < instagibbs> link to related tor source? 07:46 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev 07:49 < wumpus> https://github.com/torproject/tor/blob/master/src/or/main.c#L3210 07:50 < wumpus> https://github.com/torproject/tor/blob/master/src/common/crypto_s2k.c 07:50 < wumpus> but the python script sipa linked is more easily readable 07:50 -!- Greyboy [~Dingus@unaffiliated/greyboy] has quit [Quit: Productivity] 07:55 < sipa> oh, it has a 1-byte iteration count which uses exponential notation 07:55 < sipa> 4 bits of mantissa, and 4 bits of exponent 07:55 < wumpus> yes, the 'indicator' 07:55 < sipa> up to max 65M iterations 07:56 < sipa> and min 1024 07:56 < gmaxwell> meh. thats kinda sad. 07:56 < gmaxwell> Again I remind: so long as the format has a type field we could add other types in the future if needed. 07:57 < sipa> the tor thing actually doesn't (though it starts with an undefined "16:" prefix, which could be redefined into being a type, if necessary, i guess) 07:59 < wumpus> normally iteration is done by feeding the previous hash back into the hasher, not in this case 07:59 < wumpus> "hash the salty password as many times as the length of the password divides into the count value" 07:59 < wumpus> so it defines (roughly) how many bytes will be passed into the hash function 08:00 < wumpus> yes, the 16: in case of tor is kind of a red herring, I think anyone would have expected it to be an algorithm id 08:01 -!- MarcoFalke [8af6027e@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.126] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 08:12 -!- MarcoFalke [8af6027e@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.126] has joined #bitcoin-core-dev 08:13 < gmaxwell> wumpus: what are your thoughts on trying to make the final push to get openssl out of bitcoind? I think all code needed for it has already been written, then abandoned at varrious points in the past. The two big blockers were rpcssl and ecc, and both are resolved now. 08:14 < wumpus> we should definitely do it, but for 0.13 imo 08:14 < gmaxwell> OKAY, yes, that seems more reasonable I guess. 08:14 < sipa> agree 08:14 < wumpus> the deadline for 0.12 is getting closer and I'm sure there is high-prio stuff that still needs to get in 08:14 < gmaxwell> RNG replacement isn't a small project or one we'd want to do poorly. I think if not for that we could do the rest. 08:14 < wumpus> but as soon as 0.12 is branched off let's kill openssl 08:15 < sipa> i'll try to find the time after branching to turn the fortuna/entropy gathering code into an external library 08:15 < wumpus> I also have that on my list sipa, but feel free to do it :) 08:15 < gmaxwell> I am so tired of spurrious valgrind errors on various systems because after all these years openssl can't be bothered to not avoid undefined behavior in integrity critical RNG code. :-/ 08:16 < wumpus> yes - RNG is kind of critical in our case 08:18 < wumpus> foremost importance is that if no good entropy source can be found on a system, it's better to fail than continue with a lousy one, many wallets made that mistake 08:19 < gmaxwell> yes, if you have no good entropy and know it. Irritatingly this failure mode is not as common as it should be. :( 08:19 < sipa> "Oh, you're using Internet Explorer 3? I'll just go use { return 4; // decided by a fair dice roll } as entry! 08:19 < wumpus> exactly :D 08:19 < sipa> s/ry/ropy/ 08:19 < gmaxwell> openssl and libressl both fail that test though, even if they can't get any strong OS rng input, they keep on trucking. 08:19 < wumpus> gmaxwell: well I mean if the OS APIs somehow fail 08:21 < gmaxwell> wumpus: yea. the design Sipa had done before was also targeting improving the strength when they silently fail (happened somewhat recently in several OSes, :( ).. but certantly if the strong input fails it should full stop. 08:21 < wumpus> sure, it may happen that the OS happily returns values which happen to be perfectly predictable because it's only based on the timestamp of VM startup and timings on synthethic devices 08:22 < gmaxwell> for the GUI you can always pop up a initial start friendly game of 2048 to record user timings from; but alas, the enviroments most likely to be entropy starved are not the ones with a UI. 08:22 < wumpus> lol! 08:23 < wumpus> blockchain 2048 08:24 < wumpus> "combine the blocks to double the block reward" 08:24 < sipa> freicoin can use that 2048 version whose pawn disappear after a certain time... 08:24 < wumpus> hehehe 08:26 < gmaxwell> I've wondered what simple games extract the most entropy from users in the least time. 08:26 < sipa> gmaxwell: things with many keypresses, i'm sure :) 08:27 < gmaxwell> maybe a flashcard typing game. 08:28 < gmaxwell> oh that shows captcha flashcards so you make mistakes. :P 08:32 < wumpus> it's a nice change from the usual 'bash on the keyboard and move the mouse around like you have parkinson disease' 08:33 < sipa> LOL 08:37 < gmaxwell> wumpus: for a procedure at work I want to just instruct people to roll dice. actual dice rolls produce less entropy than the timing from typing them in, but its fun to do. 08:40 < wumpus> or combine the ideas, add a game that involves throwing dice 08:43 < wumpus> shuffling a pack of cards may also work, but it's so much work it enter it all :) 08:47 < wumpus> so sneaky, would have been so easy to miss "This should not be called with the 2 historical coinbase duplicate pairs because the new coins are marked fresh, and in the event the duplicate coinbase was spent before a flush, the now pruned coins would not properly overwrite the first coinbase of the pair. Simultaneous modifications are not allowed." 08:48 < wumpus> (re: #6932) 08:50 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev 08:51 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has joined #bitcoin-core-dev 08:53 < GitHub151> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/03403d8c0f3b...73fa5e604356 08:53 < GitHub151> bitcoin/master 14470f9 Alex Morcos: ModifyNewCoins saves database lookups... 08:53 < GitHub151> bitcoin/master 03c8282 Alex Morcos: Make CCoinsViewTest behave like CCoinsViewDB 08:53 < GitHub151> bitcoin/master 1cf3dd8 Alex Morcos: Add unit test for UpdateCoins 08:53 < GitHub31> [bitcoin] laanwj closed pull request #6932: ModifyNewCoins saves database lookups (master...newCoinsThinAir) https://github.com/bitcoin/bitcoin/pull/6932 08:55 < gmaxwell> \O/ 08:56 < sipa> someone should put a logprint in CCoinsViewDb on read and write, and reindex with max size dbcache 08:56 < sipa> there should only be 1 db operation per block left (the coinsbase overwrite) 08:57 < gmaxwell> save the logging until we've gotten that one fixed too? :) 08:58 < sipa> that needs #5967 09:01 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Ping timeout: 264 seconds] 09:06 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has joined #bitcoin-core-dev 09:07 -!- MarcoFalke [8af6027e@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.126] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] 09:22 < morcos> here you go: 09:22 < GitHub106> [bitcoin] morcos opened pull request #7056: Save last db read (master...saveLastDBRead) https://github.com/bitcoin/bitcoin/pull/7056 09:27 < gmaxwell> bam. theres to all the people shed painting over database engines, so long as you never stop your node the database no longer matters. :P 09:28 < sipa> gmaxwell: and have infinite dbcache 09:28 < gmaxwell> doesn't everyone have 128GB ram? 09:28 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev 09:30 < wumpus> gmaxwell: is the UTXO set that big now? :( 09:30 < sipa> wumpus: he runs 20 bitcoind in parallel, i'm sure 09:30 < gmaxwell> wumpus: it's about 6GB in ram right now. Less with sipa's sold script vector replacement thing. 09:30 < wumpus> hehhe 09:31 < gmaxwell> right now I have a host importing one of the satoshi dice addresses into the wallet... 09:31 < gmaxwell> this might not have been a good idea; and .. I picked one with impossibly low odds too. 09:31 -!- rubensayshi [~ruben@91.206.81.13] has quit [Ping timeout: 244 seconds] 09:32 < gmaxwell> (does anyone have an address to sugget importing that has a lot of txn but not ALL the transactions?) 09:34 < btcdrak> gmaxwell: you can find a bunch here https://www.walletexplorer.com 10:00 -!- Greyboy [~Dingus@unaffiliated/greyboy] has joined #bitcoin-core-dev 10:07 < davec> gmaxwell: if you still need one I typically use 14WCrzmqVCD7Thf79vutWTB6y1hUc8pP19 for such stress tests since it has ~62k 10:09 < gmaxwell> davec: thanks. 10:13 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev 10:16 < gmaxwell> hm. this might be a bit suboptimal; reindex with dbcach=8192. leave sitting for a day. start a wallet rescan and decide that I'd rather not, kill -9 the process. start back up with a deleted wallet.. chainstate is at height=174109 10:27 < wumpus> need more db flushes :) 10:28 < sipa> if we had flushes that didn't wipe everything, we could do them more frequently 10:28 < wumpus> I also once lost almost a whole sync in a kill -9, that's the cost of using -dbcache=8000 I guess. On the other hand it's great to be able to do a sync with almost no i/o. 10:30 < wumpus> with the standard dbcache setting it's not an issue 10:30 < gmaxwell> heh, yea, it wasn't a complaint so much as a "Oh, yea. thats ... not technically surprising." 10:32 < wumpus> does it still do a flush after IsInitialBlockDownload is complete? 10:33 < gmaxwell> I think it does, in my case I'd been reindexing and it doesn't on reindex complete AFAIK. 10:33 < wumpus> hmm maybe it should 10:42 < gmaxwell> I think so. 10:43 < wumpus> (I can't think of a reason why it'd be a bad idea to do a one-time flush after it completes indexing) 10:44 < gmaxwell> the only downside I can think of is that right now doing that will blow out the cache. 10:44 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has quit [Remote host closed the connection] 10:46 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has joined #bitcoin-core-dev 10:54 -!- Greyboy [~Dingus@unaffiliated/greyboy] has quit [Remote host closed the connection] 10:54 -!- evoskuil [~evoskuil@c-73-225-134-208.hsd1.wa.comcast.net] has joined #bitcoin-core-dev 11:15 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has quit [Ping timeout: 260 seconds] 11:45 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev 12:05 -!- trippysalmon [rob@2001:984:6466:0:a0d3:d9b8:fab7:bab3] has joined #bitcoin-core-dev 12:32 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev 13:20 -!- arubi [~ese168@unaffiliated/arubi] has quit [Ping timeout: 250 seconds] 13:28 -!- arubi [~ese168@unaffiliated/arubi] has joined #bitcoin-core-dev 13:30 < instagibbs> For debugging core where does one put the flag to do different optimization levels? 13:47 < phantomcircuit> instagibbs, CFLAGS="" ./configure 13:47 < phantomcircuit> grep for CFLAGS in config.log to see what they are by default 13:48 < BlueMatt> morcos: so I'm kinda confused about the proposed "score index" to mempool 13:48 < BlueMatt> is there any case where we really want a feerate index and do not intend to use it as "priority to get into next block" 13:48 < phantomcircuit> Luke-Jr, posted performance numbers in 6851 13:48 < BlueMatt> it seems to me we shouldnt need to indexes for this 13:48 < morcos> BlueMatt: Well it probably wont' last as its current incarnation anyway 13:49 < morcos> Why 2 indices? 13:49 < morcos> we definitely need a mining index and an eviction index? 13:49 < morcos> are those the 2 you are referring to? 13:49 < phantomcircuit> yeah see this is what i was worried about 13:49 < phantomcircuit> "just add another index" 13:50 < BlueMatt> oh, this is the inverse index of the one we have? yea, nvm, ignore me 13:50 < morcos> phantomcircuit: we right now have 3 indices. time and descendant package fee rate for eviction. and hash for lookup. 13:50 < morcos> we have to have a 4th index for mining 13:51 < morcos> because descendant package fee rate is useless 13:51 < phantomcircuit> probably dont actually need an index for time if the time based eviction has 1 hour granularity 13:51 < morcos> phantomcircuit: agreed, we could probably skip that index, and just scan the whole mempool periodically, it should be super fast 13:52 < morcos> but since it doesn't change it probably also doesnt hurt 13:52 < phantomcircuit> memory usage? 13:53 < morcos> BlueMatt: but sdaftuar was trying to add feeDelta prioritisation to your eviction code and realized that the way I did score made it hard. because he doesn't want the modified fee rate. he wants the modified fee 13:53 < morcos> So I think he will change it around, but the index will still be sorting by the same thing for mining, the internal data structure might just be a bit different 13:53 < BlueMatt> morcos: yea, fair enough...I do wonder, however, if we're gonna do a modified-feerate thing for tx selection, we probably dont want any indexes based on feerate 13:53 < BlueMatt> just base everything on modified feerate 13:54 < morcos> phantomcircuit: sure, i suppose, its really rather minor memory usage though 13:54 < morcos> BlueMatt: yes, thats what will happen 13:54 < BlueMatt> even if its modified-feereate-with-package-tracking 13:54 < BlueMatt> yea, ok 13:54 < morcos> But we need to store the actual fee without modification as well (even though there is no index on it) 13:54 < morcos> because it is now required for block construction 13:54 < morcos> so it doesn't have to be recalculated 13:56 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)] 13:56 < phantomcircuit> morcos, i assumed as much, but do we have actual numbers for it? 13:57 < morcos> phantomcircuit: memory usage? i think sipa was approximating 3 pointers per entry per index, but i'm not sure where he got that? 13:57 < morcos> sigh... just realized i forgot to fix that for my 4th index! 13:59 < morcos> sipa: is that right, was it 3 13:59 < morcos> 3 pointers per index, or did i just make that up? 14:04 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Quit: Leaving] 14:09 -!- pmienk [~pmienk@c-71-227-177-179.hsd1.wa.comcast.net] has joined #bitcoin-core-dev 14:11 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: Textual IRC Client: www.textualapp.com] 14:52 < phantomcircuit> morcos, wait why is the descendent fee index useless for mining? 14:53 < phantomcircuit> oh because you need to also track ancestor fees 14:53 < phantomcircuit> ha 15:20 -!- trippysalmon [rob@2001:984:6466:0:a0d3:d9b8:fab7:bab3] has quit [Ping timeout: 250 seconds] 15:33 < dcousens> gmaxwell: I wonder if the auth aspect will play into the RPC v REST speed, RPC can batch commands/per request which makes me think it'll lead for a while yet 15:34 < dcousens> though, without benchmarks, its hard to tell whether univalue is a factor too 15:35 < phantomcircuit> hmm 15:36 < phantomcircuit> there's no way for us to have more than 1 handle open for the berkeley db in the wallet code 15:36 < gmaxwell> univalue is pretty quick, at least my impression from AFLing it. 15:36 < phantomcircuit> so Flush is really only used for durability, right? 15:39 < phantomcircuit> ha 15:39 < phantomcircuit> i was going to ask why nMinutes is only set in CDB::Flush when we're in readonly mode, but satoshi wrote that code 15:40 < phantomcircuit> why would we be flushing the database at all if we were in read only mode? 15:40 < phantomcircuit> damn it satoshi come back so i can ask mundane questions about code 15:41 -!- andytoshi [~andytoshi@wpsoftware.net] has quit [Changing host] 15:41 -!- andytoshi [~andytoshi@unaffiliated/andytoshi] has joined #bitcoin-core-dev 15:50 -!- ParadoxSpiral [~ParadoxSp@p508B96EE.dip0.t-ipconnect.de] has quit [Ping timeout: 240 seconds] 15:59 < phantomcircuit> huh 16:10 < tulip> phantomcircuit: various religions have plans for what happens during the return of a prophet. perhaps we need a list of questions for Satoshi about coding oddities in wxBitcoin, just in case. 16:14 < phantomcircuit> wumpus, am i crazy of will CDB::Flush call bitdb.dbenv->txn_checkpoint(0,0,0) whenever fReadOnly is false 16:15 < phantomcircuit> dblogsize is ignored unless we're in read only mode 16:16 < phantomcircuit> wat 16:19 < phantomcircuit> lol no wonder this is so slow it's not flushing to disk it's reading the log files in database/* and writing them to wallet.dat 16:20 < phantomcircuit> https://docs.oracle.com/cd/E17276_01/html/api_reference/C/dbsync.html 16:21 < phantomcircuit> i wonder if that big warning means you can actually correct the database or if you will potentially lose write between write and sync calls 16:23 < phantomcircuit> im guessing the latter 16:23 < phantomcircuit> gmaxwell, ^ 16:49 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving] 17:04 -!- guest234234 [~guest2342@223.207.232.28] has joined #bitcoin-core-dev 17:14 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-zumblqpiwjatdqxa] has quit [Quit: Connection closed for inactivity] 17:23 < dcousens> gmaxwell: I'm thinking a ZMQ notification for mempool expiry would be nice 17:23 < dcousens> its kind of a hard event to catch without dumping the entire mempool list otherwise 17:23 < GitHub180> [bitcoin] pstratem opened pull request #7057: Wallet: Flush database to log files (master...2015-11-18-wallet-flush) https://github.com/bitcoin/bitcoin/pull/7057 17:23 < dcousens> do you know of a way? or is it worth me adding? 17:24 < dcousens> in general, it'd be nice to catch mempool removals 17:25 < phantomcircuit> gmaxwell, i improved my wallet performance fix to simply making the right function call in CDB::Flush 17:26 -!- vbuilderv [~JoeLiu@110.186.124.35] has joined #bitcoin-core-dev 17:35 -!- Arnavion [arnavion@unaffiliated/arnavion] has quit [Quit: Arnavion] 17:36 < GitHub41> [bitcoin] dcousens opened pull request #7058: init: amend ZMQ flag names (master...zmqdoc) https://github.com/bitcoin/bitcoin/pull/7058 17:37 -!- Arnavion [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev 17:37 < phantomcircuit> hmm there's far too few fdatasync calls in there 17:38 < phantomcircuit> i guess that means sync writes to the log but doesn't call fdatasync 17:38 < phantomcircuit> that's weird 17:39 -!- Arnavion [arnavion@unaffiliated/arnavion] has quit [Client Quit] 17:45 -!- arowser [~quassel@106.120.101.38] has quit [Quit: No Ping reply in 180 seconds.] 17:45 -!- arowser [~quassel@106.120.101.38] has joined #bitcoin-core-dev 17:58 -!- Arnavion [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev 18:21 -!- PaulCape_ [~PaulCapes@204.28.124.82] has joined #bitcoin-core-dev 18:21 -!- guest234234 [~guest2342@223.207.232.28] has quit [Ping timeout: 240 seconds] 18:23 -!- Apocalyptic [~Apocalypt@unaffiliated/apocalyptic] has quit [Ping timeout: 250 seconds] 18:23 -!- baldur [~baldur@pool-173-52-43-219.nycmny.fios.verizon.net] has quit [Ping timeout: 250 seconds] 18:23 -!- PaulCapestany [~PaulCapes@204.28.124.82] has quit [Ping timeout: 250 seconds] 18:23 -!- Eliel [~jojkaart@jkaartinen.iki.fi] has quit [Ping timeout: 250 seconds] 18:23 -!- Eliel_ [~jojkaart@jkaartinen.iki.fi] has joined #bitcoin-core-dev 18:24 -!- Apocalyptic [~Apocalypt@unaffiliated/apocalyptic] has joined #bitcoin-core-dev 18:25 -!- baldur [~baldur@pool-173-52-43-219.nycmny.fios.verizon.net] has joined #bitcoin-core-dev 18:25 < phantomcircuit> yeah ok im confused strace shows ~850 fsync/fdatasync calls with 7057 but ~30k in master 18:25 < phantomcircuit> when increasing the keypool by 10k 18:27 < phantomcircuit> (it should be called ~10k times because of the way AddKeyPubKey works) 18:42 -!- fanquake [~Adium@unaffiliated/fanquake] has joined #bitcoin-core-dev 18:43 < fanquake> jonasschnelli I'm building the gitian PR now. Just need to download some dependencies. 18:43 < fanquake> Did you build a specific version, or actually gitian build the gitian building change + master itself? 18:46 -!- challisto [~challisto@unaffiliated/challisto] has joined #bitcoin-core-dev 18:52 < dcousens> anyone here had luck with ZMQ? 18:58 < dcousens> nvm, figured, the channels are just 'hashtx', not 'pubhashtx' *whistles*, wonder if I can make that clearer somehow 18:59 < dcousens> nope, I just didn't see that part of the doc. My bad 19:18 -!- guest234234 [~guest2342@223.207.232.28] has joined #bitcoin-core-dev 20:09 < phantomcircuit> ok 7057 now does what i expected it to do --- Log closed Wed Nov 18 21:23:48 2015 --- Log opened Wed Nov 18 21:47:04 2015 21:47 -!- kanzure [~kanzure@unaffiliated/kanzure] has joined #bitcoin-core-dev 21:47 -!- Irssi: #bitcoin-core-dev: Total of 87 nicks [0 ops, 0 halfops, 0 voices, 87 normal] 21:47 -!- wangchun [~wangchun@li414-193.members.linode.com] has quit [Quit: leaving] 21:47 -!- wangchun [~wangchun@li414-193.members.linode.com] has joined #bitcoin-core-dev --- Log closed Wed Nov 18 21:55:51 2015 --- Log opened Wed Nov 18 21:57:27 2015 21:57 -!- kanzure [~kanzure@unaffiliated/kanzure] has joined #bitcoin-core-dev 21:57 -!- Irssi: #bitcoin-core-dev: Total of 87 nicks [0 ops, 0 halfops, 0 voices, 87 normal] 21:57 -!- fanquake [~Adium@unaffiliated/fanquake] has quit [Quit: Leaving.] 22:06 -!- Irssi: Join to #bitcoin-core-dev was synced in 580 secs 22:09 -!- Sanjay [~d@103.40.137.5] has quit [Ping timeout: 264 seconds] 22:23 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has quit [Quit: Lost terminal] 22:50 < GitHub150> [bitcoin] arowser opened pull request #7059: add powerpc build support for openssl lib (master...ppc) https://github.com/bitcoin/bitcoin/pull/7059 22:51 -!- challisto [~challisto@unaffiliated/challisto] has quit [Quit: Leaving] --- Log closed Wed Nov 18 22:57:33 2015 --- Log opened Wed Nov 18 22:58:43 2015 22:58 -!- kanzure [~kanzure@unaffiliated/kanzure] has joined #bitcoin-core-dev 22:58 -!- Irssi: #bitcoin-core-dev: Total of 83 nicks [0 ops, 0 halfops, 0 voices, 83 normal] 23:07 -!- Irssi: Join to #bitcoin-core-dev was synced in 574 secs 23:22 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev 23:44 -!- CodeShark [uid126576@gateway/web/irccloud.com/x-tktxznjpsgfjzthy] has quit [Quit: Connection closed for inactivity]