--- Day changed Mon Nov 16 2015
00:09 < gmaxwell> BlueMatt: did he mess up the signing? :)
00:17 < phantomcircuit> jonasschnelli, *cough*
00:18 < phantomcircuit> oh nvm ignore me
00:18 < jonasschnelli> gmaxwell: hmm... i testes the verifiy-commit stuff before. Should work if one has a recent master where he verifies commit from.
00:19 < jonasschnelli> master needs to have https://github.com/bitcoin/bitcoin/pull/7004
00:19 < jonasschnelli> then my first merge should be valid. :) (hopefully)
00:19 < phantomcircuit> jonasschnelli, it's properly signed with 32EE 5C4C 3FA1 5CCA DB46  ABE5 29D4 BCB6 416F 53EC
00:20 < jonasschnelli> ID 416F53EC, 32EE 5C4C 3FA1 5CCA DB46  ABE5 29D4 BCB6 416F 53EC 
00:20 < jonasschnelli> and this matched that: https://github.com/bitcoin/bitcoin/pull/7004/files
00:21  * jonasschnelli fingers where sweat during his first merge and double/tripple checked everything...
00:22 < gmaxwell> heh.
00:25 < wumpus> hah
00:28 < wumpus> jonasschnelli: looks good to me (w/ --show-signature)
00:29 < wumpus> (git log --merges --show-signature)
00:29 < gmaxwell> jonasschnelli: Congrats!
00:31 < wumpus> jonasschnelli: my plan was to walk you through at least one merge, but really needed some time out this weekend - seems you figured it out!
00:33 < jonasschnelli> I'm using the `github-merge.sh` script (1:1) in two other projects (https://github.com/libbtc/libbtc and https://github.com/digitalbitbox/mcu), so I'm a little bit familiar with it (also with signed commits)
00:36 < wumpus> great :D
00:38 < GitHub194> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/814697c5569c...6876a78b863e
00:38 < GitHub194> bitcoin/master 9bd3f03 Peter Todd: Clarify 'fee' field in fundrawtransaction help text...
00:38 < GitHub194> bitcoin/master 6876a78 Gregory Maxwell: Merge pull request #6991...
00:38 < GitHub101> [bitcoin] gmaxwell closed pull request #6991: Minor: Clarify 'fee' field in fundrawtransaction help text (master...clarify-fundrawtransaction-help) https://github.com/bitcoin/bitcoin/pull/6991
00:40 -!- jtimon [~quassel@74.29.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev
00:50 < GitHub34> [bitcoin] laanwj closed pull request #7021: add bip65 tests to rpc-tests.sh -extended (in 0.11 branch) (0.11...11rpcfixups) https://github.com/bitcoin/bitcoin/pull/7021
00:50 < GitHub193> [bitcoin] laanwj pushed 2 new commits to 0.11: https://github.com/bitcoin/bitcoin/compare/7e278929df53...595c8d6301cf
00:50 < GitHub193> bitcoin/0.11 9730051 Alex Morcos: add bip65 tests to rpc-tests.sh -extended
00:50 < GitHub193> bitcoin/0.11 595c8d6 Wladimir J. van der Laan: Merge pull request #7021...
00:52 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has quit [Ping timeout: 240 seconds]
00:55 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
01:13 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev
01:13 < gmaxwell> bleh, so it looks like my gbt latency measurements going up coincide not with mempool flooding attacks but with updating to master. Now I need to figure out what hurt the performance so much.
01:19 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
01:21 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
01:24 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has quit [Ping timeout: 265 seconds]
01:30 -!- rubensayshi [~ruben@91.206.81.13] has joined #bitcoin-core-dev
01:34 < wumpus> bleh indeed
01:34 < gmaxwell> https://people.xiph.org/~greg/p2pool-gbt.png < did you see my graph? :(
01:34 < wumpus> on the positive side, updating to master is easy to reproduce
01:34 < gmaxwell> well I need to figure out what I was running before! :) 
01:34 < wumpus> git reflog?
01:35 < gmaxwell> yea, I'll try (tomorrow, late here)
01:35 < wumpus> no problem, there's always tons of metadata :p
01:37 < wumpus> so it's going to be either a) the switch to libevent's http server, there may be some foul interaction with longpoll b) one of the mempool changes
01:38 < wumpus> there's also c) something other, contending the cs_main lock  .. may even be related to the pruning-related changes. Ok, conclusion: no clue.
01:38 < gmaxwell> your thinking is already much more constructive than mine, I was stuck at "oh no" :)
01:40 < gmaxwell> the GBT latency doesn't LP. it's p2pool timing how long it takes it to make a gbt call on its own. The delay is constant, so I doubt contention unless its really hot.
01:40 < gmaxwell> if it's libevent it might just be some silly tunable.
01:40 < wumpus> well it may even be just some silly mistake
01:40 < wumpus> e.g. creating an event but not triggering it
01:47 < gmaxwell> hm. I don't see the time when I moved back to test 0.11.2rc on this graph. Interesting.
01:48 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
01:48 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
01:49 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
01:49 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
01:50 < wumpus> btw using just seccomp-bpf it's not possible to limit the scope of opening files, seccomp scripts cannot dereference syscall argument pointers, it's limited to checks based on the numbers themselves  - e.g. disallow PROT_EXEC on mprotect
01:50 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has joined #bitcoin-core-dev
01:50 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has quit [Changing host]
01:50 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev
01:51 < wumpus> so if you allow open, it can open any file on the file system
01:51 < gmaxwell> oh crud, thats .. lame!
01:51 < gmaxwell> well of course selinux and such can control what files you can open.
01:52 < wumpus> it's possible to get around it with a dedicated process that traps some syscalls and then does the syscall for the restricted thread, but that's a lot more complexity
01:52 < wumpus> all those require privileged operations or changes to the OS
01:53 < wumpus> seccomp is IIRC the only one that can run unprivileged, chrome has a set-uid helper to do some jailing operations, but I don't like that
01:54 < wumpus> something may be possible with namespaces too: https://code.google.com/p/chromium/issues/detail?id=312380
01:54 < gmaxwell> Right, selinux requires privledge setup to establish the rules on the system (though after that the tags/contexts can be applied from userspace); I don't know how the apparmor stuff works.
01:54 < wumpus> apparmor works the same, it is a global config file per application
01:55 < gmaxwell> Hm I would have expected the namespaces thing to have some of the problems chroot has that makes you need privs to use it.
01:55 < gmaxwell> oh well even if we can't limit file scope at least that easily done with selinux/apparmor, and we could limit other things.
01:56 < wumpus> the advantage is that it can be set up externally, on the other hand that relies on specific files for the OS
02:00 < wumpus> nice about seccomp is that it can be set on a per-thread level. Although that doesn't help as much as long as signing and key storage happens within the main process :-)
02:01 < gmaxwell> oh I don't think I realized seccomp could be thread scoped.
02:02 < wumpus> the idea is to set up a global seccomp filter as soon as possible before any threads are created
02:02 < gmaxwell> though use of callbacks and event signaling would then make you have to be careful about what thread things actually excuted in.
02:02 < wumpus> but in a thread it can be restricted further
02:02 < wumpus> the security mesaure that amounst to is block all the exits but what people want to steal isn't outside, it's in the same memory space
02:02 < gmaxwell> e.g. I'd say the network thread could lose all file IO. ... except I'm not sure that it can.
02:03 < gmaxwell> sure if you lose control of control flow you've lost, but if the attacker can just call some libc functions, taking away file IO and such can be pretty limiting.
02:03 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
02:04 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
02:04 < wumpus> so I think we need seccomp lockdown of the main process (at least the one that does P2P networking) + key storage and signing in a different process and memory space
02:04 < wumpus> P2P networking would never have to see the passphrase or decryption key, for example
02:05 -!- CodeShark_ [~CodeShark@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
02:05 -!- CodeShark_ [~CodeShark@cpe-76-167-237-202.san.res.rr.com] has quit [Client Quit]
02:06 < wumpus> "sure if you lose control of control flow you've lost" the whole idea is to mitigate that, that if you have control flow, you can only keep up doing what the thread was doing anyway because nothing else is allowed :-)
02:06 < GitHub45> [bitcoin] jonasschnelli opened pull request #7025: [Qt] refactor and optimize proxy settings behavior (master...2015/11/qt_settingsvalidation) https://github.com/bitcoin/bitcoin/pull/7025
02:06 < gmaxwell> yea, I want to do signing in another process in any case for a multitude of reasons. Including being able to keep passphrases out of ram... being able to use HSMs, etc.
02:06 < wumpus> right
02:07 < wumpus> and my point is that I think that's a more useful step, given concrete thread models, than seccomp right now
02:07 < gmaxwell> annoyingly though, if compromising the threads that are doing network deseralization can be escaped to take over the user account, the likely they can compromise the signing process too.
02:08 < gmaxwell> Ah. Fair enough.
02:08 < wumpus> e.g. the UPnP vulnerability would have allowed getting control of a thread that has network access, no amount of lockdown couldh ave avoided it from leaking secrets from the process memory 
02:08 < wumpus> *apart* from having the secrets in a seperate process in the first place
02:08 < gmaxwell> UPNP could also be easily moved into an external program, FWIW.
02:08 < wumpus> that's not my point - UPnP is just an example. The worst case scenario would be an attack through P2P
02:08 < gmaxwell> but thats much less interesting than seperating the secrets from the attack surface, as upnp is just one vector.
02:09 < wumpus> right, better to protect what has to be protected instead
02:09 < gmaxwell> I do think if we decide we'd like to turn upnp back on, it should go in another process and _that_ one we could seccomp to have no access to open(). :)
02:10 < gmaxwell> (even if we are otherwise protecting the signing)
02:11 < wumpus> sure
02:12 < wumpus> "given concrete thread models" eh THREAT models
02:22 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
02:22 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
02:22 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
02:22 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
02:24 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
02:24 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
02:28 -!- Guest23423 [~guest2342@171.5.157.75] has quit [Ping timeout: 264 seconds]
02:37 -!- guest234234 [~guest2342@171.5.157.75] has joined #bitcoin-core-dev
02:43 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.]
02:48 < wumpus> it's possible to use unprivileged user namespaces, but only on linux kernel 3.18+
02:49 < btcdrak> the message "UpdateTip: 1 of last 100 blocks above version 4" is not correct. It's above 3.
02:51 < wumpus> BTW: if you get "/home/user/bitcoin/src/key.cpp:204: undefined reference to `secp256k1_ecdsa_sign_recoverable'"  errors after updating to master you need to clean your git tree  (since #6983)
03:02 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has joined #bitcoin-core-dev
03:03 < GitHub134> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/6876a78b863e...dafefb79244d
03:03 < GitHub134> bitcoin/master aee22bf Gregory Maxwell: Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN....
03:03 < GitHub134> bitcoin/master dafefb7 Wladimir J. van der Laan: Merge pull request #7016...
03:03 < GitHub72> [bitcoin] laanwj closed pull request #7016: Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN. (master...without_EVENT_LOG_WARN) https://github.com/bitcoin/bitcoin/pull/7016
03:03 < midnightmagic> yay!
03:23 < wumpus> user namespaces create a 'fantasy world' for a process in which it is root, but root is mapped (by the namespace) to the user own uid - it does have capabilities such as chroot *within* this user namespace, but it doesn't let it use its elevated privileges outside that. This gives me a headache.
03:23 < wumpus> "Over the years, there have been a lot of features that have been added to the Linux kernel that have been made available only to privileged users because of their potential to confuse set-user-ID-root applications.  In general, it becomes safe to allow the root user in a user namespace to use those features because it is impossible, while in a  user  namespace, to gain more privilege than the root user of a user namespace has"
03:26 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
03:28 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
03:44 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
03:45 < wumpus> not really promising regarding stable API that the example program in "man user_namespaces" is already broken in kernel 4.2.0
03:48 < wumpus> ah there was a security problem, and the kernel as well as man page was updated, but didn't have that version: http://man7.org/linux/man-pages/man7/user_namespaces.7.html#EXAMPLE
03:50 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
04:02 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
04:02 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
04:03 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Quit: Leaving]
04:04 < GitHub43> [bitcoin] MarcoFalke opened pull request #7026: [contrib] Update versionprefix to "bitcoin-core" in verify.sh (master...MarcoFalke-2015-verify) https://github.com/bitcoin/bitcoin/pull/7026
04:25 < GitHub159> [bitcoin] MarcoFalke opened pull request #7027: [qa] rpc-tests: Properly use test framework (master...MarcoFalke-2015-rpcTestCleanup) https://github.com/bitcoin/bitcoin/pull/7027
04:25 < GitHub113> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/dafefb79244d...e54ebbf60097
04:25 < GitHub113> bitcoin/master 6e18268 Pieter Wuille: Switch to libsecp256k1-based validation for ECDSA
04:25 < GitHub113> bitcoin/master e54ebbf Wladimir J. van der Laan: Merge pull request #6954...
04:25 < GitHub119> [bitcoin] laanwj closed pull request #6954: Switch to libsecp256k1-based ECDSA validation (master...secp256k1) https://github.com/bitcoin/bitcoin/pull/6954
04:25 -!- MarcoFalke [8af6026a@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.106] has joined #bitcoin-core-dev
04:28  * jonasschnelli applaud to the merge of  #6954
04:28 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has quit [Ping timeout: 250 seconds]
04:32 < jonasschnelli> wumpus: sorry for 6900, i could not create a trusty base image. Seems not to work on my debian7 machine. I tried to change the gitian vm builder... but looks like a VitrualBox mkfs issue.
04:32 < jonasschnelli> The only gitian host i have is debian... 
04:32  * jonasschnelli needs more nativ linux machines...
04:43 -!- mm_1 [bnc33@bnc33.nitrado.net] has quit [Remote host closed the connection]
04:44 -!- mm_1 [bnc33@bnc33.nitrado.net] has joined #bitcoin-core-dev
04:48 -!- ParadoxSpiral [~ParadoxSp@p508B9544.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
04:58 < phantomcircuit> wumpus, neat
05:01 < sipa> yay :)
05:02 < phantomcircuit> sipa, how long have you been working on libsecp256k1?
05:03 < sipa> i started somewhere in 2013, i think
05:03 -!- MarcoFalke [8af6026a@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.106] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
05:03 -!- MarcoFalke [8af6026a@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.106] has joined #bitcoin-core-dev
05:11 < wumpus> jonasschnelli: doesn't strictly need to be native - w/ lxc you can also use VMs, I think it's even possible to do nested kvm on recent CPUs
05:14 < btcdrak> sipa: wow, that is like a life's work
05:24 < jl2012> Sorry if this is a stupid question. Is there any contingency plan if libsecp256k1 turns out to be a softfork or hardfork?
05:30 < instagibbs> or both? ;)
05:35 < sipa> if there is any difference with openssl, it is a hard fork
05:35 < sipa> because you can negate the outcome of checksig
05:35 < sipa> so any strictening of rules can be turned into widening, in theory
05:37 < sipa> though at this point i think it's fair to say that due the amount of testing and formal validation that went into libsecp256k1, that is more likely to be a bug in openssl then
05:38 < sipa> also, it may not be exploitable... the BN_sqr bug we found in OpenSSL (thanks to libsecp256k1's unit tests that compared with OpenSSL) was technically a hard fork for bitcoin when it got foxed, but an unexploitable one as far as we know
05:39 < sipa> *fixed
05:39 < sipa> so i guess it would depend on what sort of consensus change it would be
05:42 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
05:42 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Max SendQ exceeded]
05:42 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
05:43 < sipa> and if it's before 0.12 is released, we can always just revert it :)
05:45 -!- MarcoFalke [8af6026a@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.106] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
05:53 -!- zmanian_ [uid113594@gateway/web/irccloud.com/x-gsfqmaxordvbkuwq] has quit [Ping timeout: 240 seconds]
05:54 -!- zmanian_ [uid113594@gateway/web/irccloud.com/x-bwgarzkmlsqdwohz] has joined #bitcoin-core-dev
05:54 < instagibbs> sipa, is there a summary of the review/testing process for it?
06:00 -!- MarcoFalke [8af6026a@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.106] has joined #bitcoin-core-dev
06:00 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Ping timeout: 260 seconds]
06:02 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
06:06 -!- Guest95455 [~pigeons@94.242.209.214] has quit [Ping timeout: 240 seconds]
06:07 < sipa> instagibbs: not really, but i agree we need that
06:07 -!- pigeons [~pigeons@94.242.209.214] has joined #bitcoin-core-dev
06:08 -!- pigeons is now known as Guest89319
06:08 < sipa> instagibbs: several types of tests are not yet merged (though they pass fine), including sage scripts to algebraically verify the group formulas, and an alterrnative build that results in a very small size curve so we can do high-level exhaustive testing
06:08 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
06:09 < sipa> those should go in before 0.12, as well as some document describing all the various methods
06:13 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Ping timeout: 240 seconds]
06:15 < jonasschnelli> wumpus: get a linker error when building #6900 (trusty gitian). Check: https://bitcoin.jonasschnelli.ch/pulls/6900/build-linux.log (very bottom)
06:41 < GitHub162> [bitcoin] MarcoFalke opened pull request #7028: [doc] qa: Move README.md and update -help (master...MarcoFalke-2015-qaReadme) https://github.com/bitcoin/bitcoin/pull/7028
06:54 -!- grieve [~gr1eve@50.248.81.66] has quit [Quit: Leaving]
06:58 < GitHub151> [bitcoin] sdaftuar opened pull request #7029: Remove unmaintained example test script_test.py (master...script-test-cleanup) https://github.com/bitcoin/bitcoin/pull/7029
06:58 < jgarzik> Great to see libsecp256k1 go in
06:58 < btcdrak> and openssl to be shown the door
06:59 -!- guest234234 [~guest2342@171.5.157.75] has quit [Ping timeout: 260 seconds]
07:08 < GitHub155> [bitcoin] MarcoFalke opened pull request #7030: [contrib] Delete test-patches and move testgen to devtools (master...MarcoFalke-2015-contribC) https://github.com/bitcoin/bitcoin/pull/7030
07:12 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has joined #bitcoin-core-dev
07:19 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has joined #bitcoin-core-dev
07:23 -!- fkhan [weechat@gateway/vpn/mullvad/x-iruedugszfxodfum] has quit [Ping timeout: 260 seconds]
07:33 -!- skyraider [uid41097@gateway/web/irccloud.com/x-zdvyxldrwwkbgpdv] has joined #bitcoin-core-dev
07:33 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
07:34 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
07:36 -!- fkhan [weechat@gateway/vpn/mullvad/x-ubkfcmuajrofuphz] has joined #bitcoin-core-dev
07:39 -!- bsm1175321 [~mcelrath@38.121.165.30] has joined #bitcoin-core-dev
07:44 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
07:47 < wumpus> jonasschnelli: don't really understand
07:47 < jonasschnelli> wumpus: The 6900 issues?
07:47 < wumpus> yes
07:47 < jonasschnelli> seems something with boost
07:48 < wumpus> I thought the point of gitian was that the results are the same for everyone 
07:48 < wumpus> you're also using kvm? or lxc?
07:48 < jonasschnelli> ha... right.. maybe i do it slightly different..
07:48 < jonasschnelli> wumpus: kvm,... not i try to build 0340ffb
07:49 < jonasschnelli> not/now
07:54 < wumpus> the non-working boost sleep implementation error usually means it cannot link against boost at all
07:54 < wumpus> unfortunately it's not really easy to get the config.log in the general case from the gitian builder
07:56 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
07:56 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
07:58 < wumpus> linux seems still to be building properly here - although it has the dependencies cached, could try removing the cache and starting over from the beginning but that's going to take a long time
08:13 < jonasschnelli> linux is still building here...
08:20 < jonasschnelli> wumpus : still have the linking issue on linux (https://bitcoin.jonasschnelli.ch/pulls/6900/build-linux.log)
08:21 < wumpus> can you try removing the cache? maybe something went wrong there
08:22 -!- MarcoFalke [8af6026a@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.106] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
08:22 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
08:22 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Ping timeout: 252 seconds]
08:22 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
08:27 < jonasschnelli> wumpus: remove cache = rm -Rf gitian-builder/cache/common/*?
08:28 < wumpus> not common 
08:28 < wumpus> I think only bitcoin-linux-0.12
08:28  * jonasschnelli ran rm -Rf gitian-builder/cache/bitcoin-linux-0.12/  ... does build again now
08:29 < wumpus> I think the problem in your case is that it was using the previous cache (built on precise) on trusty
08:31 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has quit [Remote host closed the connection]
08:32 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
08:32 < jonasschnelli> right. That would make sense.
08:32 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has joined #bitcoin-core-dev
08:32 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
08:34 < wumpus> going to try following gitian-building.md on a new debian8 vm
08:36 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
08:36 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
08:46 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
08:47 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
08:52 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
08:58 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has quit [Ping timeout: 264 seconds]
09:00 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
09:01 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
09:01 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
09:02 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
09:05 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Remote host closed the connection]
09:05 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
09:22 < jonasschnelli> wumpus: seems to work now: https://bitcoin.jonasschnelli.ch/pulls/6900/
09:43 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
09:43 < morcos_> sipa: Congrats and thank you and everybody who helped with libsecp256k1.  Great work!
09:44 < sipa> morcos_: thanks :)
09:46 -!- morcos_ is now known as morcos
09:56 -!- rubensayshi [~ruben@91.206.81.13] has quit [Remote host closed the connection]
10:03 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
10:04 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
10:28 -!- Guest89319 is now known as pigeons
10:30 < morcos> jonasschnelli: when you have a chance, i'd like to get your thoughts on #6134.  i want to make sure i understood your comments.
10:55 -!- PRab_ [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has joined #bitcoin-core-dev
10:56 -!- PRab [~chatzilla@2601:40a:8000:8f9b:d904:3340:2389:a85e] has quit [Ping timeout: 240 seconds]
10:56 -!- PRab_ is now known as PRab
11:12 -!- Naphex [~naphex@xotika.tv] has quit [Quit: leaving]
11:13 -!- PRab_ [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has joined #bitcoin-core-dev
11:14 -!- PRab__ [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has joined #bitcoin-core-dev
11:16 -!- PRab [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has quit [Ping timeout: 246 seconds]
11:18 -!- PRab_ [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has quit [Ping timeout: 265 seconds]
11:18 -!- PRab [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has joined #bitcoin-core-dev
11:19 -!- PRab__ [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has quit [Ping timeout: 260 seconds]
11:31 < jonasschnelli> wumpus: https://bitcoin.jonasschnelli.ch/pulls/6900/build-win.log MinGW still has troubles with boost #6900
11:31 < jonasschnelli> ah. wait. I forgot to clean the cache for windows. Nm.
11:31 < jonasschnelli> morcos: sure. 
11:31 < jonasschnelli> Let me open the PR first
11:34 < phantomcircuit> 2015-11-16 19:34:00 PartitionCheck : Found 24 blocks in the last 4 hours
11:34 < phantomcircuit> 2015-11-16 19:34:00 PartitionCheck : likelihood: 0.0811515
11:34 < phantomcircuit> that seems like a small window
11:37 < jonasschnelli> morcos: I have to admit, that i haven't fully reviewed #6134, my focus was the GUI part.
11:37 < morcos> jonasschnelli: i just wanted to make sure i understood your comment about never returning -1
11:38 < jonasschnelli> I think for an user it's strange that he might get the same fee for targeting conf. withing the next 25blocks and the next 18blocks.
11:38 < jonasschnelli> morcos: I was calling estimatefee instead of estimateaproxfee
11:38 < morcos> part of the point of the PR was to make it so you got -1 a lot less often.  but i didn't change what estimatefee returns, i just made a new function estimateapproxfee which is smarter
11:38 < morcos> perhaps i should call it estimatesmartfee instead
11:39 < morcos> that one will still return -1, but only at the very startup when there is not enough data period (and you don't have an old estimates file, although thats maybe dangerous for its own reasons)
11:39 < jonasschnelli> I think the name is okay. I just wasn't really concentrated while reviewing and mixed up the extimateaproxfee and estimatefee
11:39 < morcos> ok
11:39 < morcos> as for same fee for different confirmations
11:39 < jonasschnelli> UI: why can't you target for 1 block?
11:40 < jonasschnelli> (conf. withing next block)
11:40 < jonasschnelli> *within 
11:40 < morcos> i'm not really sure there is anything to be done about that.  its just as accurate an answer as you can get from the data
11:40 < morcos> how do you mean, you can target for 1 block
11:40 < jonasschnelli> The slider i got was from block 25 to block 2.
11:41 < jonasschnelli> So, as enduser i would whish i could select 1 block.
11:41 < morcos> ok, so that is the UI change i wanted fee back from
11:41 < morcos> if you look at the right end of the slider
11:41 < morcos> position 1 and position 2, both say 2 blocks
11:42 < jonasschnelli> Right. I saw that. Is that because 1 block is just very hard (impossible to estimate)?
11:42 < morcos> so you are actually trying to get an estiamte for 1 block, but estimatefee returns -1.   so estimateapproxfee looks at 2 blocks, gives you an answer there, and lets you know that the answer is for 2, so you're not expecting to get confirmed in 1 with that fee
11:42 < jonasschnelli> * ... hard (impossible) to estimate?
11:42 < morcos> yes, thats correct, there is not enough data to indicate that any fee will have very high likelihood of being confirmed in 1 block
11:43 < morcos> this is primarily because i require 95% (used to be 85%) of txs with that fee or higher to be confirmed within the desired number of blocks, and often miners only recalculate their block every minute
11:43 < morcos> so that puts a ceiling of about 90% on the chance you will be confirmed in the next blcok
11:44 < morcos> my prior version of the change had the slider snap back to position 2, but that seemed like i did not implement that very well.  so the simpler change is this, but i'm happy for suggestions on how to improve it from a UI standpoint
11:45 < morcos> what i'm trying to communicate is we can't give you an answer for 1 block, the best i can do is give you an answer for 2 blocks and that answer is X
11:45 < jonasschnelli> hm... hmm.. somehow it feels that the user must be able to select 1 block... i know, it's not possible to estimate... :)
11:46 < jonasschnelli> we could also go the way other wallets do: "confirmation speed" : "slow" = ~8b | "avg" ~4b | "fast" ~2b.
11:47 < jonasschnelli> we just use the words "slow", "average", "fest"
11:47 < jonasschnelli> *"fast"
11:47 < morcos> i do think the ability to choose any # is overkill
11:47 < morcos> but maybe thats too big a change now for 0.12?
11:48 < jonasschnelli> I mean, it's nitpicking. The change loos good to me (need to test more in detail).
11:48 < jonasschnelli> *looks
11:48 < jonasschnelli> what do you think by aggregating block/fees.
11:48 < morcos> ok...  what do you think about changing estimateapproxfee to estimatesmartfee.  i think i like that name better
11:49 < morcos> better holds to how i think that function will evolve in the future
11:49 < jonasschnelli> yeah. Smart* sounds always good. I also like estimatesmartfee better.
11:49 < morcos> aggregating block/fees?
11:49 < jonasschnelli> the thing is, if you move the slider, you might get the same fee between block 25 and block 18 (was the case in my test),... 
11:50 < jonasschnelli> we could just stop at block 18 (as min value, to very left).
11:50 < morcos> hmm.. the problem is its not really possible to know any 2 targets would give you the same number until you ask
11:51 < jonasschnelli> we could run from 2-25, then aggregate values, invalidate (this cache) after a new block comes in (or similar)
11:51 < jonasschnelli> but, yeah. sounds after some lines of code.
11:51 < jonasschnelli> could also be done later.
11:51 < jonasschnelli> I mean the PR should not be hold back because of the UI!
11:51 < morcos> yeah i think after we get 0.12.  we should talk about how to improve it
11:52 < morcos> maybe do something like only store results for 1,2,4,8,16  (and then possibly add longer time horizons: 32,64,128,256?)
11:52 < morcos> and then thats the only place the slider stops
11:52 < morcos> perhaps they will sometimes give the same value, but a lot less often
11:52 < jonasschnelli> yes. This could make sense.
11:53 < jonasschnelli> IMO we should keep the UI as it is for #6134 and focus on get it in.
11:53 < jonasschnelli> What is the reason to have two estimatefee rpc calls. Keep the API?
11:53 < morcos> ok agreed.  especially as i am out of town for 10 days starting on friday!
11:53 < morcos> yeah i think fee estimation still has some evolving to do
11:54 < jonasschnelli> Take your laptop with you. :)
11:54 < morcos> so i'd like to have estimatesmartfee be the new API, that will evolve with the functionality
11:54 < morcos> but for the meantime keep the old API consistent until we have a sense of where the smart version will end up
11:55 < jonasschnelli> will it only be estimation? Or would a RPC call "smartfee estiamte" make more sense?
11:55 < morcos> i didn't understand
11:55 < jonasschnelli> *estimate
11:56 < jonasschnelli> I'm just thinking loud. Maybe other smartfee interaction would be possible in future, setting some parameters or similar.
11:56 < morcos> oh, like you could configure state within the estimator, like smartfee timehorizon 1008 
11:56 < jonasschnelli> Then we could collect everything in the smartfee rpc call.
11:57 < morcos> well i think maybe i should just call it smartfee and smartpriority ?
11:57 < jonasschnelli> something like "smartfee estimate" and "smartfee set timehorizon 100"
11:57 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has quit [Ping timeout: 240 seconds]
11:58 < morcos> i'm not sure we'll want to set state like that, i think it would be safer to only be able to configure parameters on startup
11:58 -!- AtashiCon [arnavion@unaffiliated/arnavion] has quit [Disconnected by services]
11:58 -!- Arnavion3 [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev
11:58 -!- Arnavion3 is now known as AtashiCon
11:58 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Read error: Connection reset by peer]
11:58 < jonasschnelli> Hm.. no. better keep it "estimatesmartfee".
11:59 < morcos> ok...  estimatesmartfee it is.  i'll rebase at some point with just the name change...
11:59 < jonasschnelli> We could still have something like "estiamtesmartfee settimehorzon X"
11:59 < jonasschnelli> I'll try to do a full test after your rebase.
11:59 < morcos> ok
11:59 < jonasschnelli> Keep the UI stuff as it is. We can polish that later.
12:01 < morcos> yikes, so many commits, i'm just going to make one final commit which changes the names i think...
12:01 < jonasschnelli> Ok. 
12:02 < jonasschnelli> morcos: one thing: I think we sould remove ""\nWARNING: This interface is unstable and may disappear or change!\n"" 
12:02 < morcos> oh why?  i wanted to have that in there so people didn't write code depending on estimatesmartfee
12:02 < morcos> i'm keeping the estimatefee interface the same
12:03 < morcos> and for the most part they can design their own smart algorithm by using estimatefee directly
12:03 < morcos> but i don't want to tie down the interface to the built-in smart algorithm now
12:03 < morcos> i almost didn't expose it as rpc calls, but its nice for testing
12:03 < jonasschnelli> Well, i think if we pack that into a release, we need to consider that people write code depending on that... new features in a 0. version are not meant to habe stable API IMO
12:04 < jonasschnelli> *have
12:04 < morcos> well thats the point of the warning, so they don't write code or they're aware.  i think that was wumpus' recommendation
12:04 < jonasschnelli> But no strong opinion on that.
12:04 < jonasschnelli> Okay. Then lets keep this.
12:04 < morcos> ok thx
12:05 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #bitcoin-core-dev
12:06 < jonasschnelli> morcos: nice code/PR by the way. You guys always come directly with a rpc test and a bunch of unittests!
12:06 < morcos> ha, sdaftuar makes me!
12:06 < jonasschnelli> hah!
12:06 < jonasschnelli> Good boy.
12:07 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has joined #bitcoin-core-dev
12:42 -!- belcher [~user@94.10.240.224] has joined #bitcoin-core-dev
12:43 -!- belcher is now known as Guest49481
12:43 -!- Guest49481 [~user@94.10.240.224] has quit [Remote host closed the connection]
12:45 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev
13:00 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
13:04 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
13:05 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
13:17 -!- d_t [~textual@c-50-136-139-144.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
13:32 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev
13:37 < gmaxwell> https://www.reddit.com/r/Bitcoin/comments/3t04fp/psa_please_upgrade_to_bitcoin_core_0112_to/cx2b117 < anyone know whats up with someone saying they can't upgrade bitcoin core without upgrading OSX?
13:45 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)]
14:11 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has joined #bitcoin-core-dev
14:11 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has quit [Changing host]
14:11 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev
14:12 < treehug88> gmaxwell : he doesn't mention which OS version he's currently on. I had no trouble upgrading to 0.11.2 on 10.10.5 (Yosemite)
14:18 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: Textual IRC Client: www.textualapp.com]
14:19 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Quit: Leaving]
14:32 -!- ParadoxSpiral [~ParadoxSp@p508B9544.dip0.t-ipconnect.de] has quit [Remote host closed the connection]
14:52 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Quit: This computer has gone to sleep]
14:54 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Ping timeout: 265 seconds]
14:54 -!- bsm117532 [~mcelrath@static-108-21-236-13.nycmny.fios.verizon.net] has quit [Ping timeout: 265 seconds]
14:54 -!- gmaxwell [greg@wikimedia/KatWalsh/x-0001] has quit [Ping timeout: 265 seconds]
14:54 -!- bsm1175321 [~mcelrath@38.121.165.30] has quit [Ping timeout: 265 seconds]
14:54 -!- berndj [~berndj@azna.co.za] has quit [Ping timeout: 265 seconds]
14:54 -!- isis [~isis@abulafia.patternsinthevoid.net] has quit [Ping timeout: 265 seconds]
14:55 -!- gmaxwell [greg@mf4-xiph.osuosl.org] has joined #bitcoin-core-dev
14:55 -!- gmaxwell is now known as Guest42878
14:55 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has joined #bitcoin-core-dev
14:56 -!- Guest42878 [greg@mf4-xiph.osuosl.org] has quit [Changing host]
14:56 -!- Guest42878 [greg@wikimedia/KatWalsh/x-0001] has joined #bitcoin-core-dev
14:56 -!- Guest42878 is now known as gmaxwell
14:59 -!- berndj [~berndj@azna.co.za] has joined #bitcoin-core-dev
15:00 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Disconnected by services]
15:03 -!- isis [~isis@abulafia.patternsinthevoid.net] has joined #bitcoin-core-dev
15:05 -!- wump [~quassel@pdpc/supporter/professional/wumpus] has joined #bitcoin-core-dev
15:06 -!- Netsplit *.net <-> *.split quits: wumpus
15:07 -!- bsm117532 [~mcelrath@static-108-21-236-13.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
15:09 -!- zooko [~user@2601:281:8001:26aa:8874:a77d:6fe6:611e] has joined #bitcoin-core-dev
15:09 -!- bsm1175321 [~mcelrath@38.121.165.30] has joined #bitcoin-core-dev
15:09 -!- guest234234 [~guest2342@223.207.202.174] has joined #bitcoin-core-dev
15:10 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #bitcoin-core-dev
15:13 -!- guest234234 [~guest2342@223.207.202.174] has quit [Ping timeout: 240 seconds]
15:27 -!- zarathustra [zanoni@znc.openshells.net] has joined #bitcoin-core-dev
15:27 -!- zarathustra [zanoni@znc.openshells.net] has quit [Changing host]
15:27 -!- zarathustra [zanoni@unaffiliated/tolstoi] has joined #bitcoin-core-dev
15:48 -!- dcousens [~anon@c110-22-219-15.sunsh4.vic.optusnet.com.au] has joined #bitcoin-core-dev
15:52 -!- skyraider [uid41097@gateway/web/irccloud.com/x-zdvyxldrwwkbgpdv] has quit [Quit: Connection closed for inactivity]
15:53 < dcousens> sipa: so, :S
15:53 < dcousens> I have about 10mib garbage at the end of my leading .dat atm
15:53 < dcousens> and one of my .dat files in the middle has 23mb garbage at the end
15:54 < dcousens> other than that, all blocks accounted for
15:54 < dcousens> bitcoind's rescan went without a hitch
16:03 < dcousens> also, unrelated, but the total parse time (I realised that, before due to lazy evaluation, it was skipping ahead), when the result is piped to /dev/null, its parsing at ~210MiB/s, otherwise its blocked by output to about 110MiB/s
16:04 < dcousens> (meaning, about 7 minutes for a full 50G parse w/ ~50G output)
16:05 -!- guest234234 [~guest2342@171.5.156.180] has joined #bitcoin-core-dev
16:16 < dcousens> sipa: I also found a blog post detailing what I'm running into, tl;dr all the garbage is just a long run of 0's
16:16 < sipa> dcousens: that's preallocation
16:17 < dcousens> sipa: is it meant to be written though?
16:17 < sipa> dcousens: we preallocate the block files when first creating them, and then overwrite pieces of it with blocks; at the end, when leaving the file, we truncate
16:18 < dcousens> sipa: well, considering my bitcoind is closed, and the file still has a huge run of 0's at the end
16:18 < sipa> but things can go wrong i guess (for example, if the first write after restart is to a new file, you never 'leave' the old file)
16:18 < dcousens> I'm guessing the truncate isn't 100% solid 
16:18 < sipa> dcousens: the last file will always have zeroes at the end
16:18 < sipa> as it isn't full yet
16:18 < dcousens> ok, by leaving the file you meant moving onto the next
16:18 < sipa> right, indeed
16:20 < gmaxwell> we prefill with zeros to prevent fragmenting the crap out of non-fragmentation resistant file systems with our incremental writes.
16:20 < dcousens> gmaxwell: gotcha
16:21 < dcousens> though, it probably shouldn't be in the middle of a .dat though
16:25 < dcousens> a 10MiB gap infact, then 252 more blocks after
16:27 < gmaxwell> dcousens: are these old files? We previously had bugs that might cause behavior like that, but I think they're all fixed.
16:27 < GitHub175> [bitcoin] ptschip opened pull request #7034: Fixed Windows secp256k1 compilation issue #7018 (master...secp256k1_compile) https://github.com/bitcoin/bitcoin/pull/7034
16:27 < dcousens> gmaxwell: its a fresh IBD with a just-compiled 4 days ago bitcoin/bitcoin master
16:28 < dcousens> at most the build would be within 2 weeks
16:28 < gmaxwell> okay, that sounds like a bug then. 
16:29 < dcousens> Happy to donate time to debug it, but, not sure where to start :S
16:29 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
16:30 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
16:35 < GitHub69> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/e54ebbf60097...0a547d2d5501
16:35 < GitHub69> bitcoin/master 4d29032 Eric Lombrozo: Fixed integer comparison warning.
16:35 < GitHub69> bitcoin/master 0a547d2 Gregory Maxwell: Merge pull request #7023...
16:35 < GitHub106> [bitcoin] gmaxwell closed pull request #7023: Fixed integer comparison warning. (master...signed_int_comparison_fix) https://github.com/bitcoin/bitcoin/pull/7023
16:39 < GitHub66> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/0a547d2d5501...972bf9c52987
16:39 < GitHub66> bitcoin/master f6d9d5e Jonas Schnelli: add (max)uploadtarget infos to getnettotals RPC help
16:39 < GitHub66> bitcoin/master 972bf9c Gregory Maxwell: Merge pull request #6999...
16:40 < GitHub149> [bitcoin] gmaxwell closed pull request #6999: add (max)uploadtarget infos to getnettotals RPC help (master...2015/11/maxuploadtarget_rpc_help) https://github.com/bitcoin/bitcoin/pull/6999
16:41 < GitHub25> [bitcoin] ptschip closed pull request #7034: Fixed Windows secp256k1 compilation issue #7018 (master...secp256k1_compile) https://github.com/bitcoin/bitcoin/pull/7034
16:48 -!- Guest23423 [~guest2342@171.5.156.180] has joined #bitcoin-core-dev
16:48 -!- zooko [~user@2601:281:8001:26aa:8874:a77d:6fe6:611e] has quit [Ping timeout: 240 seconds]
16:52 -!- guest234234 [~guest2342@171.5.156.180] has quit [Ping timeout: 264 seconds]
16:54 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-aimtdiiuklyzfhvg] has quit [Quit: Connection closed for inactivity]
17:00 < GitHub18> [bitcoin] gmaxwell pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/972bf9c52987...87ee0e2dbc12
17:00 < GitHub18> bitcoin/master 598e494 Jorge Timón: Chainparams: Explicit CChainParams arg for main (pre miner):...
17:00 < GitHub18> bitcoin/master 6bc9e40 Jorge Timón: Chainparams: Explicit CChainParams arg for miner:...
17:00 < GitHub18> bitcoin/master 87ee0e2 Gregory Maxwell: Merge pull request #6986...
17:00 < GitHub68> [bitcoin] gmaxwell closed pull request #6986: Globals: Don't call Params() from miner.cpp (master...global-chainparams-miner) https://github.com/bitcoin/bitcoin/pull/6986
17:13 < GitHub122> [bitcoin] dcousens opened pull request #7035: torcontrol: only output disconnect if -debug=tor (master...torlog) https://github.com/bitcoin/bitcoin/pull/7035
17:15 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
17:15 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
17:15 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has joined #bitcoin-core-dev
17:16 < gmaxwell> wumpus: so valgrind is telling me the event_new TorController::disconnected_cb is creating a leak on clean shutdown with stop.  But I've confirmed the destructor that calls event_del is correctly firing.  Is there some extra libevent magic needed to cause this to get freed? 
17:16 < dcousens> gmaxwell: wumpus isn't here? :S
17:16 < sipa> it seems he got truncated into wump
17:17 < gmaxwell> I wondered why my client completed 'wump' :) (but apparently I didn't wonder that much)
17:37 -!- Guest23423 [~guest2342@171.5.156.180] has quit [Ping timeout: 250 seconds]
17:49 < morcos> gmaxwell: This is not a complaint but just a question, because I do think we should merge more of Jorge's refactoring changes, but we planning on trying to find a time to do a bunch of them so that we can consolidate the merge conflicts?
17:53 < gmaxwell> Feel free to complain in any case; I was mindful of it creating some conflict-- but it didn't move any code and I thought refactors that created structural changes were the big concern there.  I looked at existing open PRs (including yours) and concluded that the disruption from this would be small. (and if I felt that I could have merged yours first now and asked Jorge to rebase, I would have d
17:53 < gmaxwell> one that).
17:55 < morcos> yeah like i said, i'm not complaining, rebasing #7020 took a second, and #6898 is going to need some more work depending on #7020 and #7008 anyway.  But I just wanted to be aware if we were about to have a slew of these, since we're also trying to get a bunch of stuff in before the freeze.
17:55 < gmaxwell> morcos: if you go and fix your pull req, please provide feedback by screaming my name on IRC in proportion to what a pain in the ass iti is. :)
17:56 < gmaxwell> Ah, no I wasn't planning on pulling most of the other ways. (there was another very minor change from jtimon that I just asked him to rebase wrt the one I just merged, that I think won't break anything)
17:56 < morcos> speaking of, #6898 , the CreateNewBlock pull, it is quite a risky change, although with a big benefit.  What else do you think we should do to get comfortable with it.
17:57 < morcos> For instance, I called it out in the initial PR comment, but I didn't realize until reading your comments on IRC this weekend that not stopping to add txs at some minimum feerate is actually a pretty big negative change
17:58 < morcos> I've since fixed that...  But it worries me to change the code that miners everywhere will be relying on without extensive testing.   What should that be?
18:03 < gmaxwell> A couple things: we can probably add some tests that hand it simulated mempools and check some additional invarients, like that the selection is feerate monotone.  We can deploy it early 'mining' (doesn't have to start actually mining, just call createnewblock and sanity check the output) on the main network. We can probably get some miners with non-trival hashpower to deploy it post merge durin
18:03 < gmaxwell> g the 0.12 testing cycle; allowing us to catch an extreme stupidity before release.
18:04 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving]
18:04 < gmaxwell> It's also bad news that we have so many speedups in 0.12. That means miners will likely upgrade very quickly.
18:04 < sipa> morcos: would it be possible to add the new getblocktemplate as a new RPC (getblocktemplatebeta or so) ?
18:05 < sipa> and leaving the old one as-is for now
18:05 < gmaxwell> Otherwise we would have some buffering from "well, it'll take a while for everyone to deploy it."
18:05 < gmaxwell> hm. if we do that I'd want to do something to identify the blocks created with it... but I don't know that we can. :(
18:06 < sipa> it probably makes little difference
18:07 -!- zooko [~user@c-73-217-16-2.hsd1.co.comcast.net] has quit [Ping timeout: 244 seconds]
18:07 < gmaxwell> well I just mean that if we see miners making pathalogically stupid blocks it would be nice to be able to tell if the new code is to blame. :)
18:08 < sipa> gmaxwell: i mean that miners will likely use it anyway, even if it's called beta (although that does mean changing some software...)
18:09 < morcos> sipa: i'm sure thats easy enough to do.  Maybe that makes sense in that if there is a problem its easy to revert back without having to stop using 0.12
18:09 < dcousens> a rescan shouldn't modify the .dat files, should it?
18:09 < gmaxwell> sipa: that could also be a flag: oldcreatenewblock=0 
18:10 < sipa> dcousens: no
18:10 < gmaxwell> Another reason to keep the old one around is if the miners are carrying patches to the old one, they can upgrade to 0.12 and keep them in place... OTOH, more for us to test.
18:10 < morcos> I'd tested previously that it generated the exact same blocks, modulo the comments I put at the top of the pull.  But I did forget one more.  I'll go update that.  I stop trying to fill the last little spot in the block early, but if I increase how much it'll search for that , they were the same
18:12 < morcos> I think I like this idea though.  If its a command line option then all we need is my duplicated CreateNewBlock code , and getblocktemplate checks the argument to see which CreateNewBlock it should call?
18:12 < phantomcircuit> gmaxwell, the coinbase sig flags returned by getblocktemplate could be modified
18:12 < phantomcircuit> im not sure if any miners actually use that though
18:12 < morcos> Hmm..  kind of annoying for unit tests and rpc tests though....
18:13 < phantomcircuit> when is feature freeze for 0.12 ?
18:13 < sipa> morcos: or CreateNewBlock itself dispatches based on the command-line flag?
18:13 < gmaxwell> phantomcircuit: lots of people ignore the coinbaseflags coming out of gbt. :(
18:14 -!- guest234234 [~guest2342@171.5.156.180] has joined #bitcoin-core-dev
18:15 < gmaxwell> morcos: it could just be a commandline flag, doubt we want rpc callers picking it on the fly.
18:15 < morcos> One other comment I'll make about 6898 is that its clearly a WIP in the larger sense too.  If we can rip out priority for 0.13, it'll clean up the code a lot, and also we might rework it yet again to do some intelligent CPFP
18:16 < morcos> so we should think a bit about how we might wan tto make future major changes to the code as well?
18:17 < morcos> gmaxwell: yes thats what i meant (checks the command line argument) i agree you should not be able to toggle back and forth
18:17 < gmaxwell> Well I do kind of like having the ability to swap in new algorithims with runtime selection.
18:17 < gmaxwell> I don't know how realistic that is generally, since I think different algorithims will require different indexes for efficient implementation.
18:18 < morcos> yes and keeping two algorithms properly tested is basically impossible
18:20 < gmaxwell> well right now with the current one there is a montone property I think it will obey, in that it will always select the highest feerate transactions, so anything omitted must be equal to or lower than the bottom feerate. So testing on real and simulated loads will at least tell us if its behaving crazily (like failing to mine a particular transaction causing it to clog the mempool)
18:20 < gmaxwell> But fancier algorithms will lose that simple property.
18:20 -!- guest234234 [~guest2342@171.5.156.180] has quit [Read error: Connection reset by peer]
18:34 < morcos> gmaxwell: you're assuming wiht a priority space of 0?  Also that's not exactly correct, in that a high fee rate child could only have become elgible to be included when there was no longer enough space for it.
18:35 < gmaxwell> Yes, that was assuming priority space of 0. 
18:35 < morcos> But I was suggesting we hack up the existing code to change the tie breaker and require that the blocks are identical for testing.  I think that's easy enough to achieve.  The problem is whether there is some degenerate situation where something bad happens
18:35 < gmaxwell> ah, indeed, it still needs a topological sort.
18:36 < phantomcircuit> morcos, so the answer i would like to give is "it doesn't matter they can run both and the block verification at the end will protect them from bugs"
18:36 < phantomcircuit> but i know that lots of them dont have proper setups so that isn't really true
18:36 < phantomcircuit> :(
18:37 < gmaxwell> phantomcircuit: if you note above I mentioned the case of some high feerate txn not getting mined when it should.
18:38 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Read error: Connection reset by peer]
18:39 < morcos> So there are some very weird edge case differences I guess.  For instance, one of the versions has the sigop check before the block priority space check and the other has it after.  so if you have a transaction that fills the block priority space, you now switch to fee rate sorted, but if it fails the sigop test, then code A will have moved on to fee txs, but code B will still try for another priority tx
18:40 < morcos> i can't remember which is which, and i haven't encountered that yet, nor do i expect to...   but at a certain point, mimicking the old behavior is not what we want
18:41 < gmaxwell> no reason to think much of the old behavior is any good.
18:43 -!- zooko [~user@2601:281:8001:26aa:8874:a77d:6fe6:611e] has joined #bitcoin-core-dev
18:45 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has joined #bitcoin-core-dev
18:57 < phantomcircuit> gmaxwell, i assume that was from a block maxing out checksigs?
19:32 < dcousens> gmaxwell: "One thought I had on this (before realizing there was already a pool"
19:32 < dcousens> did you mean s/pool/pull/?
19:36 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has quit [Read error: Connection reset by peer]
19:38 -!- davec [~davec@cpe-24-243-251-52.hot.res.rr.com] has joined #bitcoin-core-dev
19:51 -!- jtimon [~quassel@74.29.134.37.dynamic.jazztel.es] has quit [Ping timeout: 250 seconds]
19:58 < BlueMatt> gmaxwell: nope, actually, master still verifies with all the pgp sigs
20:02 < BlueMatt> ofc jonasschnelli's key is completely unsigned, so......
20:02 < gmaxwell> dcousens: yep
20:02 < sipa> BlueMatt: i'll go see him when i'm in .ch (assuming he doesn't hate my face)
20:26 < BlueMatt> sipa: sign wump's key while you're up north :p
20:52 -!- Squidicuz [~squid@pool-173-48-117-206.bstnma.fios.verizon.net] has joined #bitcoin-core-dev
20:56 -!- sipa_ [~pw@2a02:348:86:3011::1] has joined #bitcoin-core-dev
20:56 -!- cfields_ [~quassel@unaffiliated/cfields] has joined #bitcoin-core-dev
20:59 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has quit [Read error: Connection reset by peer]
21:00 -!- jonasschnelli [~jonasschn@unaffiliated/jonasschnelli] has quit [Ping timeout: 246 seconds]
21:00 -!- zmanian_ [uid113594@gateway/web/irccloud.com/x-bwgarzkmlsqdwohz] has quit [Ping timeout: 246 seconds]
21:00 -!- Squidicc [~squid@pool-173-48-117-206.bstnma.fios.verizon.net] has quit [Ping timeout: 246 seconds]
21:00 -!- da2ce7 [~da2ce7@opentransactions/dev/da2ce7] has quit [Ping timeout: 246 seconds]
21:00 -!- sipa [~pw@2a02:348:86:3011::1] has quit [Ping timeout: 246 seconds]
21:00 -!- andytoshi [~andytoshi@unaffiliated/andytoshi] has quit [Ping timeout: 246 seconds]
21:00 -!- teward [teward@ubuntu/member/teward] has quit [Ping timeout: 246 seconds]
21:00 -!- cfields [~quassel@unaffiliated/cfields] has quit [Ping timeout: 246 seconds]
21:00 -!- phantomcircuit [~phantomci@strateman.ninja] has quit [Ping timeout: 246 seconds]
21:00 -!- amiller_ [~socrates1@li175-104.members.linode.com] has quit [Ping timeout: 246 seconds]
21:00 -!- BlueMatt [~BlueMatt@unaffiliated/bluematt] has quit [Ping timeout: 246 seconds]
21:00 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has joined #bitcoin-core-dev
21:00 -!- Guest802 [~socrates1@li175-104.members.linode.com] has joined #bitcoin-core-dev
21:00 -!- jonasschnelli [~jonasschn@2a01:4f8:200:7025::2] has joined #bitcoin-core-dev
21:00 -!- phantomcircuit_ [phantomcir@2600:3c01::f03c:91ff:fe73:6892] has joined #bitcoin-core-dev
21:02 -!- BlueMatt [~BlueMatt@unaffiliated/bluematt] has joined #bitcoin-core-dev
21:02 -!- zooko [~user@2601:281:8001:26aa:8874:a77d:6fe6:611e] has quit [Ping timeout: 240 seconds]
21:03 -!- zmanian_ [sid113594@gateway/web/irccloud.com/x-yxnanrfkmjkkhadp] has joined #bitcoin-core-dev
21:03 -!- andytoshi [~andytoshi@wpsoftware.net] has joined #bitcoin-core-dev
21:05 -!- teward [teward@ubuntu/member/teward] has joined #bitcoin-core-dev
21:09 -!- Guest802 [~socrates1@li175-104.members.linode.com] has quit [Changing host]
21:09 -!- Guest802 [~socrates1@unaffiliated/socrates1024] has joined #bitcoin-core-dev
21:09 -!- Guest802 is now known as amiller
21:10 -!- da2ce7 [~da2ce7@opentransactions/dev/da2ce7] has joined #bitcoin-core-dev
21:15 -!- phantomcircuit_ is now known as phantomcircuit
21:17 -!- zmanian_ [sid113594@gateway/web/irccloud.com/x-yxnanrfkmjkkhadp] has quit [Ping timeout: 246 seconds]
21:18 -!- btcdrak [uid115429@gateway/web/irccloud.com/x-yvkcjomrigurxopx] has quit [Ping timeout: 250 seconds]
21:20 -!- CodeShark [uid126576@gateway/web/irccloud.com/x-qjfqykpchqoysfzj] has quit [Ping timeout: 250 seconds]
21:25 -!- Netsplit *.net <-> *.split quits: amiller, zarathustra
21:26 -!- amiller_ [~socrates1@li175-104.members.linode.com] has joined #bitcoin-core-dev
21:29 -!- CodeShark [uid126576@gateway/web/irccloud.com/x-estmhssftbtzwzlu] has joined #bitcoin-core-dev
21:35 -!- btcdrak [uid115429@gateway/web/irccloud.com/x-ycavaifvoqdrsxim] has joined #bitcoin-core-dev
21:37 -!- zmanian_ [uid113594@gateway/web/irccloud.com/x-hajqnxerbyvtyssb] has joined #bitcoin-core-dev
22:07 -!- zarathustra [zanoni@znc.openshells.net] has joined #bitcoin-core-dev
22:07 -!- zarathustra [zanoni@znc.openshells.net] has quit [Changing host]
22:07 -!- zarathustra [zanoni@unaffiliated/tolstoi] has joined #bitcoin-core-dev
22:07 -!- pigeons [~pigeons@94.242.209.214] has quit [Ping timeout: 250 seconds]
22:08 -!- pigeons [~pigeons@94.242.209.214] has joined #bitcoin-core-dev
22:08 -!- pigeons is now known as Guest96832
22:12 < dcousens> sipa_: so
22:12 < dcousens> I'm still debugging this, and the garbage in my blk00289
22:13 < dcousens> is random transactions, but its not at all an actual block
22:13 < dcousens> like, its not prefixed by a header
22:14 < dcousens> http://pastie.org/10561905
22:15 < dcousens> Any ideas?
22:16 < dcousens> that pastie is hard to decipher, but tldr I printed some bytes from the previous block to see if maybe the length was off, but, again, that block verifies so :/
22:16 < dcousens> no idea, maybe theres a loose buffer overwrite somewhere?
22:17 < gmaxwell> dcousens: while syncing, we write to the file and only flush the indexes periodically, if you shut off prematurely, it'll continue at the last index write, and overwrite whatever blocks were there. Might this be consistent with what you're seeing?
22:18 < dcousens> gmaxwell: so you're thinking it had written a block, of say size N, it was then shutdown prematurely, and on re-open, it then wrote block of size M at that same index, where M < N?
22:20 < gmaxwell> Yes. 
22:20 < gmaxwell> though unless it were the last block placed in the file (according to the block index), I don't know why it wouldn't have overwrote the lest.
22:21 < midnightmagic> out-of-order block storage
22:21 < midnightmagic> makes sense.
22:22 < midnightmagic> sort of..
22:29 < dcousens> gmaxwell: if I nuke this file
22:29 < dcousens> and rescan
22:29 < dcousens> Will it survive? :P
22:30 < gmaxwell> it'll rescan up to that part and begin downloading the rest after it.
22:30 < gmaxwell> (by rescan I mean reindex)
22:30 < dcousens> uh yeah
22:30 < dcousens> reindex*
22:30 < dcousens> won't use any of the other .dat files?
22:30 < dcousens> (as in, it'll abandon all blocks after it and re-download the remaining 15G?)
22:31 < gmaxwell> it'll use all the earlier ones, and none of the later ones.
22:32 < dcousens> ok
22:32 < gmaxwell> It doesn't have a sophicated indirect membership management because ... that only makes sense if you're going to do crazy things like randomly delete a file in the middle. :)
22:33 < dcousens> gmaxwell: :), well, at this point I can do this, or just ignore bytes until I reach a magic int then verify the block :/
22:34 < dcousens> the 2.5 minute blockchain parse is too good to give up now haha
22:35 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
22:35 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
22:36 < gmaxwell> dcousens: what are you doing this parse for?
22:37 < dcousens> gmaxwell: custom index building for various dbs, its the fastest way to bootstrap, 5minutes to write it all out then just use the RPC to get anything else
22:37 -!- tulip [~tulip@128.199.135.43] has joined #bitcoin-core-dev
22:38 < dcousens> much nicer then waiting 40+hrs for an RPC level sync, or hacking bitcoind to do it for me
22:38 < gmaxwell> You benchmark that rpc scan since we got libevent? 
22:38 < dcousens> Its just become a bit of a timesink now because of this weird run of 000's and this issue.
22:39 < dcousens> gmaxwell: actually I haven't
22:40 < gmaxwell> I don't know which changes made it faster, but its much faster than it used to be; still going to be slow compared to scanning the files directly probably.
22:40 < dcousens> gmaxwell: but maybe not slow enough to warrant this, point
22:40 < gmaxwell> (well also, if its slow but fast enough to use; I'd like to optimize it further!)
22:41 < dcousens> gmaxwell: the point was also so I had a tooling to be able to do things like check ancestor/depths super quick etc
22:42 < dcousens> for service level bootstrapping, the new RPC speed may be enough,but analytically I'd still like a fast parse.
22:43 < gmaxwell> just take care, there have been some things that lost funds because they were doing block scanning and didn't know some blocks were orphans... :-/
22:44 < dcousens> gmaxwell: indeed, noticed the orphans
22:44 < tulip> there's also traps for people trying to do block parsing as well, some transactions contain blocks. 
22:45 < dcousens> haha, like an 80byte OP_RETURN?
22:45 < tulip> the minimum block size is more than 80 bytes, but something like that. 
22:48 < dcousens> gmaxwell: I'll give a play at the RPC sync speed
22:48 < dcousens> let you know :)
22:59 < gmaxwell> anyone around that understands the ZMQ interface? I'm trying to figure out how it hooks in to get a signal of a new block.  I see nothing in the codebase that references CZMQNotificationInterface::UpdatedBlockTip at all.
23:00 < dcousens> gmaxwell: give me an hour and I might be sharing that question with you
23:04 < aj> gmaxwell: src/validationinterface.cpp sets it up as a signal, and src/main.cpp triggers the signal?
23:05 < aj> gmaxwell: or src/init.cpp has RegisterValidationInterface(pzmqNotificationInterface); to set it up in the first place?
23:10 < gmaxwell> thanks! 
23:12 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-aebyobfgbxedgxua] has joined #bitcoin-core-dev
23:13 < GitHub177> [bitcoin] gmaxwell opened pull request #7037: Move the blocknotify callback ahead of peer announcement. (master...notify_early) https://github.com/bitcoin/bitcoin/pull/7037
23:15 -!- wump is now known as wumpus
23:29 < GitHub2> [bitcoin] gmaxwell pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/87ee0e2dbc12...eac53ec99201
23:29 < GitHub2> bitcoin/master 141c44e MarcoFalke: [contrib] Update versionprefix to "bitcoin-core" in verify.sh
23:29 < GitHub2> bitcoin/master a6d5a65 MarcoFalke: [trivial] contrib: Fix `echo`s in verify.sh
23:29 < GitHub2> bitcoin/master eac53ec Gregory Maxwell: Merge pull request #7026...
23:29 < GitHub74> [bitcoin] gmaxwell closed pull request #7026: [contrib] Update versionprefix to "bitcoin-core" in verify.sh (master...MarcoFalke-2015-verify) https://github.com/bitcoin/bitcoin/pull/7026
23:33 < gmaxwell> Luke-Jr: https://github.com/bitcoin/bitcoin/pull/6851 wants rebase
23:42 -!- MarcoFalke [c3523fc5@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.197] has joined #bitcoin-core-dev
23:51 -!- jonasschnelli [~jonasschn@2a01:4f8:200:7025::2] has quit [Changing host]
23:51 -!- jonasschnelli [~jonasschn@unaffiliated/jonasschnelli] has joined #bitcoin-core-dev