--- Day changed Wed Oct 21 2015
00:00 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has quit [Ping timeout: 252 seconds]
00:04 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
00:05 < phantomcircuit> BlueMatt, great now i can start reviewing it
00:05 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has joined #bitcoin-core-dev
00:05  * phantomcircuit flees for his life
00:05 < BlueMatt> phantomcircuit: Ive done that before :/
00:08 < wumpus> gmaxwell: would be interesting to check at least; switching to tcmalloc is as simple as LD_PRELOAD a library, don't know about jemalloc
00:09 < gmaxwell> same thing for jemalloc.
00:09 -!- Arnavion [arnavion@unaffiliated/arnavion] has quit [Ping timeout: 252 seconds]
00:10 -!- AtashiCon [arnavion@unaffiliated/arnavion] has quit [Ping timeout: 250 seconds]
00:11 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has quit [Quit: quit]
00:12 < phantomcircuit> i wonder how much it would mess up things to put all the global config things into a locked object
00:13 < wumpus> good as a workaround but I'd prefer a solution that doesn't involve making changes at that level. C++ makes it possible to use own memory pools/allocators for specific things, that can be discarded in one block (say) at the end of the function, that seems to be a more predictable way
00:15 < gmaxwell> I wish the standard API just let you hint short and long lived objects.
00:15 < wumpus> phantomcircuit: which things aren't protected by mutexes that should be ?
00:15  * phantomcircuit looks at gmaxwell 
00:16 < wumpus> at the least you'd want a configuration object per module, not single global one, there is already too much lock centralization
00:19 < phantomcircuit> wumpus, i believe we do bad things with stuff like fReindex and nPruneTarget
00:23 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has joined #bitcoin-core-dev
00:29 < wumpus> gmaxwell: yes, unfortunately C and C++ are very much lacking with regard to giving hints to your compiler or runtime, it presupposes them to be much more important
00:29 < wumpus> s/important/smart
00:51 < btcdrak> gmaxwell: what are the deployment options for SG?
00:51 < btcdrak> err, SW.
00:51 < gmaxwell> btcdrak: a flag day swap of the network and all software to use a different transaction seralization. 
00:52 < gmaxwell> I don't think it can be done.
00:52 < btcdrak> gmaxwell: if we're going to have a hard fork, we may as well take the opportunity then...
00:52  * btcdrak runs
00:52 < phantomcircuit> btcdrak, there's a huge list of things that should absolutely be in a flag day
00:54 < btcdrak> SW is a big win, solves a lot of problems and would make a flagday worth the hassle. If we do end up with a blocksize increase, SW should be added at the same time imo.
00:54  * wumpus throws a hard frok after btcdrak
00:55 < Luke-Jr> gmaxwell: I thought I had convinced sipa a softfork was possible for SW a week or two ago. Or is that just too ugly?
00:55  * btcdrak has never tried to cross-dress before
00:56 < phantomcircuit> btcdrak, the blocksize thing is spv backwards compatible, the only other hard fork proposal im aware of that is, is for commitment schemes in the merkle tree root
00:56 < phantomcircuit> note: i dont actually see that as being very useful
00:57 < btcdrak> Luke-Jr: I'm sure I had a conversation with someone recently, maaku or BlueMatt maybe, about how SW could be a soft fork.
00:57 < phantomcircuit> spv clients can all be updated remotely by the devs since they're apps in an app store thingie
00:57  * Luke-Jr agrees SPV-backward-compatible is not a particularly desirable attribute.
00:58 < phantomcircuit> SW cannot really be a soft fork, you can commit to the sw mtr as a sf but that only gets you the uninteresting part of it
00:58 < phantomcircuit> oh but actually you can force the tx to commit to the sw txid in the script sig
00:58 < phantomcircuit> ha you could but... please no
00:59 < Luke-Jr> phantomcircuit: would it really be worse than this thing cdecker is proposing?
00:59 < gmaxwell> btcdrak: it cannot be a softfork. only a weak approximation of it can be, which is what the decker email is.  But that approximation doesn't get you benefits like being able to sync blocks without downloading signatures. (itself likely elimiating most of the need for bloom filtered lite wallets, among other benefits);  ... and the approximation has costs like a considerable utxo size overhead, a
00:59 < gmaxwell> nd doubling the amount of transaction hashing we have to do.
01:00 < Luke-Jr> gmaxwell: I don't see why you can't sync without downloading signatures in the softfork.
01:00 < gmaxwell> Luke-Jr: because you can't verify a txout you learn without the whole transaction. 
01:01 < gmaxwell> (unless in a softfork you mean extensionblock like softforks)
01:02 < Luke-Jr> gmaxwell: the witness wouldn't be part of the "whole transaction"; it would be stored in a separate piece of data the block commits to
01:02 < btcdrak> SW was quite well received in this thread: https://www.reddit.com/r/Bitcoin/comments/3ngtx5/could_the_segregated_witness_part_of_the/
01:03 < phantomcircuit> block commits to the normal txid as normal and the sw id & script hash separately
01:03 < gmaxwell> Luke-Jr: hm. interesting, you have a point. So you'd basically have a scriptpubkey that says "go look elsewhere for the signature"
01:03 < Luke-Jr> right
01:04 < gmaxwell> still ends up needing to store two IDs.
01:04 < gmaxwell> hm. or maybe it doesn't.
01:05 < phantomcircuit> gmaxwell, segregated witness id should match transaction id if you simply require all the scripts in the normal transaction to be OP_1
01:05 < gmaxwell> Luke-Jr: damn you're right. This is pretty cool.
01:05 < gmaxwell> phantomcircuit: yea.
01:06 < Luke-Jr> ☺
01:06 < gmaxwell> phantomcircuit: empty is better.
01:07 < gmaxwell> So the idea there would be a new p2sh like scriptpubkey which requires the scriptsig to be empty.  The signatures are 'external', and commited in another hashtree commited to by the block.
01:08 < gmaxwell> it's identical to SW over whatever span of tx history is exclusively using this scriptpubkey type..
01:09 < Luke-Jr> (which could be enforced by the softfork, if it was desirable)
01:10 -!- rubensayshi [~ruben@91.206.81.13] has joined #bitcoin-core-dev
01:11 < gmaxwell> Luke-Jr: then it turns it into a flagday which is the hard thing to avoid.
01:11 < Luke-Jr> sure
01:11 < Luke-Jr> personally I think we might as well just do it in a hardfork
01:11 < Luke-Jr> but a softfork is possible
01:12 < gmaxwell> the problem is that it is not very reasonable to demand the entire bitcoin ecosystem cut cold to new code all at once.
01:12 < gmaxwell> (I mean, you could try to demand it but the result would be epic amounts of failure)
01:12 < gmaxwell> esp at the high degree that people reimplement everything themselves.
01:12 < Luke-Jr> we'd need to ship a version with both code
01:12 < Luke-Jr> but that's true of any hardfork
01:13 < gmaxwell> Sure, yea yea, the issue isn't us. it's j random wallet thingy that hardly works at all _currently_.  It might even manage to ship code for both modes, but then the other one just won't actually work.
01:13 < Luke-Jr> oh, you mean tx creation :D
01:13 < gmaxwell> because outside of bitcoin core most software (but by no means all) in the ecosystem is developed by teams of one or two people, and shipped with little to no systematic testing.
01:14 < gmaxwell> not just creation but handling.
01:14 < gmaxwell> Not how we still have no remotely decent block explorer thingy for elements alpha.
01:14 < Luke-Jr> yeah, I guess that's a big problem
01:15  * Luke-Jr ponders if we can do the softfork way that remains tx-compatible with a future hardfork
01:18 < gmaxwell> I think it's almost that already, without special considerations.
01:18 < Luke-Jr> maybe
02:01 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev
02:20 < GitHub36> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/3b20e239c602...0fbfc5106cd9
02:20 < GitHub36> bitcoin/master 41db8c4 Wladimir J. van der Laan: http: Restrict maximum size of request line + headers...
02:20 < GitHub36> bitcoin/master 0fbfc51 Wladimir J. van der Laan: Merge pull request #6859...
02:20 < GitHub199> [bitcoin] laanwj closed pull request #6859: http: Restrict maximum size of http + headers (master...2015_10_max_http_headers) https://github.com/bitcoin/bitcoin/pull/6859
02:58 < gmaxwell> Is there a reason that for fee purposes we are not using max(size, sigops*BLOCK_MAX_BYTES/BLOCK_MAX_SIGOPS) as the size of a transaction?
02:58 < GitHub192> [bitcoin] domob1812 opened pull request #6863: [Test Suite] Fix test for null tx input (master...null-txin-test) https://github.com/bitcoin/bitcoin/pull/6863
03:02 < gmaxwell> I'd like to see this go in https://github.com/bitcoin/bitcoin/pull/6622  I've had this in testing on several nodes (including one acting as a gateway to the outside world) for several others for over a month now.
03:02 < gmaxwell> Does anyone have any views on what else we need? for #6622?
03:07 < gmaxwell> libevent related compile fail on one of my older fedora hosts,   error: ‘EVENT_LOG_WARN’ was not declared in this scope
03:17 < btcdrak> gmaxwell: if this is the first time compiling since libevent merge you need to do a git clean -dfx
03:21 < gmaxwell> btcdrak: fwiw, in my case no concern but you might want to take care in advising people to git clean -d :)
03:21 < gmaxwell> "omg I was keeping my wallet in my source directory!"
03:25 < wumpus> at least mention what it does
03:26 < gmaxwell> yea.
03:26 < gmaxwell> So in any case, cleaning doesn't make that go away. I see some other EVENT logging stuff appears to be ifdef guarded.
03:26 < wumpus> but after larger changes to the build system it's good advice
03:27 < gmaxwell> this appears to have libevent-2.0.so.5.1.6
03:28 < wumpus> I don't know what the minimum version of libevent is that is supported, may be too old
03:29 < gmaxwell> well hacking out that one like got it to build.
03:29 < wumpus> though if it just errors about the logging stuff, commenting that out may make it compile further
03:29 < wumpus> good
03:29 < gmaxwell> er one line.
03:30 < gmaxwell> testing now.
03:30 < gmaxwell> warren: what libevent version does current RHES have?
03:30 < wumpus> it's not necessary, it's just nice, redirects libevent errors to our debug log, likely you could put some #ifdef guard around it...
03:30 < gmaxwell> unit tests pass. 
03:31 < wumpus> #if LIBEVENT_VERSION_NUMBER >= 0x....
03:36 < warren> gmaxwell: RHEL7 you mean?
03:36 < gmaxwell> warren: Yes, whatever version is most current.
03:37 < warren> libevent-2.0.21-4.el7
03:37 < warren> Keep in mind that if <whatever> is important you need to actually look at the source package as they often will forward port things.
03:37 < gmaxwell> warren: okay, thats newer than what this host has in any case.
03:38 < warren> wow, that's the same version in Fedora 22.  I guess it didn't change upstream in a while.
03:39 < gmaxwell> the particular host I was trying on is my oldest still running fedora box, with F19.
03:40 -!- Naphex [~naphex@unaffiliated/naphex] has quit [Quit: leaving]
03:42 -!- Naphex [~naphex@xotika.tv] has joined #bitcoin-core-dev
03:42 -!- Naphex [~naphex@xotika.tv] has quit [Changing host]
03:42 -!- Naphex [~naphex@unaffiliated/naphex] has joined #bitcoin-core-dev
03:47 < wumpus> 2.0.21 is the one-to-latest stable of libevent, they have been working on 2.1 for a long time but it's still in alpha
03:58 < GitHub79> [bitcoin] MarcoFalke opened pull request #6864: [qt] Use monospace font (master...MarcoFalke-2015-qtMonospace) https://github.com/bitcoin/bitcoin/pull/6864
04:13 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.]
04:40 < btcdrak> gmaxwell: omg
05:03 -!- Arnavion [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev
05:03 -!- AtashiCon [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev
05:45 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
05:47 -!- molly [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
05:48 -!- ParadoxSpiral [~ParadoxSp@p508B9A93.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
05:53 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
06:04 -!- berndj [~berndj@azna.co.za] has joined #bitcoin-core-dev
06:06 < jgarzik> gmaxwell, ok to merge #6622, IMO
06:13 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Ping timeout: 264 seconds]
06:24 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
06:33 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.]
06:33 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
06:38 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 268 seconds]
06:51 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
07:00 -!- wangchun [~wangchun@li414-193.members.linode.com] has quit [Remote host closed the connection]
07:22 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
07:26 -!- ghtdak [~ghtdak@unaffiliated/ghtdak] has quit [Quit: WeeChat 1.4-dev]
07:38 -!- ghtdak [~ghtdak@unaffiliated/ghtdak] has joined #bitcoin-core-dev
07:41 -!- MarcoFalke [8af602a9@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.169] has joined #bitcoin-core-dev
07:43 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
07:52 -!- MarcoFalke [8af602a9@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.169] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
07:52 -!- MarcoFalke [8af602a9@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.169] has joined #bitcoin-core-dev
07:55 -!- sipa [~pw@2a02:348:86:3011::1] has joined #bitcoin-core-dev
08:27 -!- MarcoFalke [8af602a9@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.169] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
08:47 -!- wangchun [~wangchun@li414-193.members.linode.com] has joined #bitcoin-core-dev
08:48 -!- wangchun [~wangchun@li414-193.members.linode.com] has quit [Client Quit]
08:48 -!- wangchun [~wangchun@li414-193.members.linode.com] has joined #bitcoin-core-dev
09:02 -!- rubensayshi [~ruben@91.206.81.13] has quit [Ping timeout: 240 seconds]
09:27 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: Textual IRC Client: www.textualapp.com]
09:47 -!- jl2012 [~jl2012@unaffiliated/jl2012] has joined #bitcoin-core-dev
10:19 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has joined #bitcoin-core-dev
10:19 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has quit [Changing host]
10:19 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev
10:28 -!- Anduck [~anduck@unaffiliated/anduck] has quit [Ping timeout: 240 seconds]
10:28 -!- Anduck [~anduck@unaffiliated/anduck] has joined #bitcoin-core-dev
10:29 -!- teward [teward@ubuntu/member/teward] has quit [Ping timeout: 240 seconds]
10:34 -!- CodeShark__ [CodeShark@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
10:37 -!- teward [teward@ubuntu/member/teward] has joined #bitcoin-core-dev
10:39 < gmaxwell> cfields_: Any thoughts on #6622?  If you look at it-- it's largely orthorgonal with your overall networking work, happening at a higher level. (later rate limiting support would probably compliment it nicely)
10:44 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Read error: Connection reset by peer]
10:44 < cfields_> gmaxwell: hmm
10:46 < gmaxwell> wow, we never went and set TCP_NODELAY on our sockets? we should do that right away-- perhaps even in backports.  We've talked about it before but apparently never did it.
10:46 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #bitcoin-core-dev
10:50 < jgarzik> gmaxwell, not that I disagree, but... what's your specific motivation to use NODELAY & disable Nagel?
10:52 < gmaxwell> jgarzik: now that we've removed all the gratitious sleeps from the networking, nagle is almost certantly slowing our performance. Consider how chatting the bitcoin protocol is.  It likely also makes the traffic more bursty, which isn't good for other users sharing the network.
10:53 < gmaxwell> Basically all of our writes should be large enough that nagle shouldn't be saving us much, and the smaller writes we to tend to be latency important.
10:53 < jgarzik> gmaxwell, nod
10:53 < jgarzik> gmaxwell, just theory or is someone reporting this?
10:53 < jgarzik> certainly appears low hanging fruit
10:54 < cfields_> gmaxwell: agree that the logic for deciding what to avoid sending belongs at a higher level. The implementation itself seems a bit naive, though?
10:54 < gmaxwell> jgarzik: Suhas mentioned in an email some testing that seemed impacted by it.  I'm just surprised we hadn't done this already, historical analysis suggests it's somewhat my fault! :(
10:55 < sdaftuar> jgarzik: yeah tcp_nodelay speeds up block relay by a round trip
10:55 < gmaxwell> (jgarzik you brought it up previously, and I (thinking of the 100ms sleeps) said we had much more low hanging fruit. :P )
10:56 < jgarzik> yep it was an issue when I rewrote low level network reads/writes code
10:56 < gmaxwell> I'd previously gone around to miner software and p2pool and had them fix it, even before it had been brought up in bitcoin core.
10:56 < jgarzik> thanks.  I was curious where it would first trip up ppl
10:56 < btcdrak> can someone with rights please restart this travis job please? https://travis-ci.org/bitcoin/bitcoin/jobs/86431798
10:58 < cfields_> btcdrak: just force-push
10:58 < btcdrak> not my PR
11:00 < morcos> wumpus: sipa: i've been trying to learn a bit more about this intensive memory usage in getblocktemplate.  i still haven't quite figured out why the framentation is so bad that nothing can get clean up, but there are a few things happening
11:00 < morcos> during CreateNewBlock, we fetch all the coins needed for all the txs in our mempool.
11:01 < morcos> there is a 2x hit for this, as we make a copy in pcoinstip (assuming they aren't already cached there) and a copy in the view in miner.cpp
11:02 < morcos> also note that the dbcache size we set here will have just been ignored as we're loading up pcoinstip.  so if we're over that it'll just get flushed again once we're done with CNB
11:02 < morcos> also because the rpc call is running in a different thread, if the populating of the cache in pcoinstip trips the loadfactor of the CCoinsMap.  Then we'll end up moving the CCoinsMap memory to a different arena.   
11:03 < wumpus> right, that is how it will happen with the coinsview caches
11:03 < morcos> so both the size of the coins in the miner.cpp view and the size in the coins of pcoinstip could possibly be replicated in #rpcthreads + 1
11:04 < wumpus> indeed
11:04 < gmaxwell> morcos: sipa and BlueMatt have been been working on the view cache memory usage.
11:04 < morcos> i'm curious as to what they're doing
11:04 < wumpus> it seems a bit redundant that the child cache also stores everything
11:04 < jgarzik> morcos, RE fragmentation part of that has to do with the low level allocator that how that memory is returned to the OS.  Older allocators use sbrk(), which make it impossible to release memory unless there is absolutely no data structure, including hidden-from-app mgmt ADTs, within the memory range.  Ditto newer mmap-based allocators, which will not munmap unless there is perfect reclamation by app + libc.
11:04 < morcos> yes, thats what i was thinking
11:05 < jgarzik> So you're stuck even if libc allocation structs are left behind
11:05 < jgarzik> (typically two pointers, in negative offsets from your DS)
11:05 < morcos> wumpus
11:06 < wumpus> especially as this is a read-only view
11:06 < morcos> wumpus: which is the child cache?  The cache in view has to store everything b/c it might get modified by in mempool txs
11:06 < wumpus> oh it isn't?
11:06 < morcos> however the cache in pcoinstip doesn't need to be populated
11:06 < morcos> view (in miner.cpp) is backed by pcoinstip is backed by the database
11:06 < wumpus> that's the parent cache
11:07 < wumpus> and sure - the modified coins need to be stored
11:07 < morcos> i think its kind of silly that you set a dbcache size, but then can blow through that inside CreateNewBlock
11:08 < morcos> of course its also made much worse by the organization of the mining code
11:08 < morcos> no reason to go through every single tx
11:09 < wumpus> e.g. the coins that are the same in the child and parent cache don't need to be stored twice
11:09 < wumpus> only if the one in the child cache starts to deviate because ti is changed by a mempool transaction
11:09 < morcos> yes, thats what i'm suggesting.. when you do a fetch from the child cache, just pass them down from the grandparent cache without storing them intermediately in the parent cache
11:10 < wumpus> right, assuming they aren't changed in the parent cache, they don't need to be cached there
11:10 < morcos> this is related to #5967
11:11 < morcos> where i change the assumptions that the parent cache can't be flushed 
11:13 < morcos> slight shift of subject, how do most miners use bitcoind now?
11:13 < morcos> they do use getblocktemplate?
11:14 < wumpus> is there another way?
11:14 < morcos> and in effect have CreateNewBlock control their mining algorithm, or they do something else?
11:14 -!- CodeShark_ [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Remote host closed the connection]
11:14 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Quit: AndroIRC - Android IRC Client ( http://www.androirc.com )]
11:14 -!- CodeShark__ [CodeShark@cpe-76-167-237-202.san.res.rr.com] has quit []
11:14 < morcos> i don't know i thought i saw chatter somewhere about miners not using it
11:14 -!- CodeShark [CodeShark@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
11:14 < morcos> but i dont' knwo what else you would do
11:14 < wumpus> I think they all use getblocktemplate in some form or another, although some may have made large changes to the code
11:14 < morcos> i'm wondering how to think about rewriting the mining code
11:15 < morcos> it could use a substantial rewrite
11:15 < morcos> but i guess it doesn't have to be thought of as consensus critical if the testing/submitting aspects of it haven't changed?
11:15 -!- CodeShark_ [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
11:16 < wumpus> block validation is consensus critical, creating blocks from the mempool isn't
11:16 < gmaxwell> morcos: I suspect you misunderstood whatever that chatter was (or someone was confused); everyone uses GBT to get work from bitcoind.  Many parties also do other stuff as well.
11:16 < gavinand1esen> morcos: +1 . I'd suggest starting with a clean sheet of paper, and figuring out what would be best for solo miners / mining pools
11:16 < morcos> it seems like there is some low hanging fruit.  once you already have 999,900 bytes in a block, you don't need to scan the remaining 2GB of txs just to see if you find a magical 100 byte tx for instance
11:16 < morcos> gmaxwell: ok good... i definitely didn't understand whatever i read
11:16 < gmaxwell> so long as the resulting block is valid I don't think it matters how it was done; so I think it's fine to rewrite.
11:17 < gmaxwell> morcos: might have been a comment that few/no one is using GBT to go from pools to mining devices; which is the case.
11:17 < wumpus> yeah it obviously matters that it generates correct blocks, but the exact algorithm is not set in stone
11:18 < gmaxwell> morcos: as far as rewrite goes, being able to get createnewblock out of the latency critical path for mining would be the biggest win from the user's perspective.
11:18 < morcos> does it make sense to start with a simple improvement to the existing algorithm, and then separately or later try to use more advanced logic that takes ancestor packages into account?  would it make sense to ever support 2 different algorithms
11:19 < morcos> gmaxwell: can you explain that a bit more?  you mean once a new block comes in, the time it takes to have a new potential header to work on?
11:19 < gmaxwell> (e.g. by returning an empty template when there isn't one precomputed, then computing a template in the background.).. and thats independant of the algorithims, but it means that more computationally expensive algorithims would be reasonable.
11:19 < morcos> even if its not optimal
11:19 < morcos> yes, ok that makes a lot of sense
11:20 < gmaxwell> morcos: what I was thinking was: there is a cached block template, when a new block comes in, it's no longer valid (as we have a new block) so its flushed. If there is no cached template available, just generate one that has no transactions (fast), and trigger creating a template in the background.... and the background one can get recomputed on a timer or on new-block arrival, so long as GBT re
11:20 < gmaxwell> quests keep coming in.
11:20 < gmaxwell> we cache CNB right now, but it's "too late" :)
11:21 < jgarzik> +1
11:21 < jgarzik> that's been the general template for a rewrite, for years
11:22 -!- Guest98736 is now known as petertodd
11:22 < morcos> gmaxwell: yes, thats what we were just discussing... 
11:22 -!- CodeShark_ [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Remote host closed the connection]
11:25 < morcos> so a new thread for template generation is ok?
11:26 < gmaxwell> morcos: I think so!
11:26 < jgarzik> practically required
11:26 < wumpus> I don't see why not, but make sure to only do the work if a node is actually mining
11:27 < morcos> what should a call to getblocktemplate do if cs_main is locked?  seems too bad that you really dont' care for a lot of the cs_main locks
11:27 < morcos> but if the chain is being updated, you might prefer to wait for that to happen before the template is returned
11:27 < jgarzik> does it need cs_main if it's simply returning a cached version?
11:28 < jgarzik> seems like the RPC call should return cached version, and b/g compute thread is what needs the lock
11:28 < morcos> i suppose maybe you could lock the template if you know you're activating a new chain
11:28 < morcos> ok i guess i need to learn more about how notification of new tips works anyway
11:28 < jgarzik> if a new block comes in, the cache is invalidated + regenerated as an empty template + kicks off new gen thread
11:28 < morcos> but yes i agree in theory you should just need a tiny lock on whether your template is switched to the new one
11:29 < gavinand1esen> I was very close to refactoring the block validation code a tiny bit so the cs_main lock was released while transaction validity was checked... theoretically not a difficult change
11:29 < morcos> gavinand1esen: that seems scary
11:29 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Ping timeout: 264 seconds]
11:29 < gavinand1esen> that's why theoretically....
11:29 < jgarzik> should just need a quick swap upon new-block, not a long term lock
11:30 < morcos> jgarzik: isn't the technical word for that "tiny"?
11:30 < jgarzik> can be lockless technically :)
11:31 < gmaxwell> yes, the cache could be RCU, but I don't think thats needed. :)
11:31 < jgarzik> I was thinking more std::atomic
11:33 < morcos> ok all sounds good... i'll give it a shot
11:34 < gavinand1esen> morcos: I was mapping out the work needed to validate blocks in parallel (as part of prep work for looking at broadcasting 'weak blocks'); was pondering an extension to CCoinsViewCache that used leveldb read-only snapshots to validate against the UTXO set as of the last block, which could let validation against two different CCoinsViewCache's happen in parallel without the cs_main lock
11:35 < gavinand1esen> morcos: that idea might work for a cs_main-free CreateNewBlock, too....
11:35 -!- gavinand1esen is now known as gavinandresen
11:36 < sipa> signature validation already happens without the cs_main lock
11:41 < morcos> sipa: you mean b/c they are happening in other threads?  but the cs_main lock is held and any further processing waits for them to finish
11:42 < morcos> sipa: btw, did you see my comments about doing threaded signature checking in ATMP in addition to in ConnectBlock?
11:42 < morcos> it seems thats when the actually work is being done a lot of the time anyway
11:44 < sipa> morcos: i did not see that, but it certainly may make sense
11:44 < sipa> i'd like to move signature checking to be something fully asynchronous though
11:45 < sipa> with a queue that is worked on by threads, and notification callbacks happen when validatiin succeeds/fails
11:46 < morcos> oh...  huh, so a block could just assume that all signatures were valid and keep processing the rest of the blcok, but only at the very end wait
11:46 < morcos> wow that would be way better
11:46 < sipa> it's nontrivial :)
11:46 < morcos> not for you!
11:46 < jcorgan> i think it would be a great idea...if sipa does it :-)
11:46 < gmaxwell> morcos: yes, ... behold the sadness of CHECKSIG NOT, however.
11:47 < morcos> hmm... so you don't even know whether you want it to be valid or not
11:47 < gmaxwell> if all of script is async then that goes away.
11:47 < morcos> i thought processors were good at branching. :)
11:48 < morcos> what do you mean goes away?
11:48 < gmaxwell> I mean that if the thing you dispatch is the whole script and not just ecdsa then there is no chance that you really want it to fail.
11:48 < jgarzik> processors try to predict branching because they suck at it ;p
11:48 < morcos> oh... 
11:48 < gmaxwell> but then the state that needs to be dispatched is large.
11:49 < jgarzik> The kernel compiles Berkeley packet filters in a JIT - would be amusing to JIT scripts & sigs
11:49 < gmaxwell> jgarzik: by amusing you mean the tremendous fun in finding all the attacks...
11:49 < gmaxwell> :P
11:50 < gmaxwell> morcos: in Elements alpha we turned CHECKSIG into CHECKSIGVERIFY  (if you want a checksig thats allowed to fail, wrap it in an if).
11:50 < sipa> gmaxwell: i mean parallellizing script validation
11:51 < sipa> not signature validation
11:51 < gmaxwell> sipa: k. then the negation problem goes away... but lots of data to manage.
11:52 < gmaxwell> (the CHECKSIGVERIFY vs CHECKSIG is also motivated by batch EC signature checking, which is faster and only gives you a pass/fail for the whole batch).
11:52 < morcos> i'd feel better about if we more formally declared what the state that can affect script validation is
11:52 < morcos> yeah do we need CHECKSIG?
11:54 < gmaxwell> No, there is no need for CHECKSIG--- in theory we could softfork convert existing checksig into a CHECKSIGVERIFY; though some risk of breaking some crazy existing scriptpubkey and making it unspendable.
11:54 < gmaxwell> So thats probably not advisable, but any future checksig like operators will certantly be VERIFY only.
11:55 < morcos> wouldn't that not be a soft fork because of the CHECKSIG NOT's
11:55 < morcos> it seems like you could just bump tx version if you didn't want to worry about existing scriptpubkeys
11:56 < gmaxwell> morcos: it would be a soft fork: If you had a CHECKSIG NOT with a bad input  you'd just get denied before you got to the NOT.
11:57 < gmaxwell> Less agressive but probably equally useful is CHECKSIG requiring that the only signature that is allowed to fail is the zero length signature. (so you can decide if it fails or not without doing anything but checking the length)
11:58 < gmaxwell> morcos: key criteria for a soft-fork is that it cannot turn any invalid thing valid. So a soft fork that just added additional reasons for the script to fail when excuting a checksig would be a softfork (if not an advisable one!)
11:58 < gmaxwell> the only-zero-length can fail also has the advantage of being less potentially incompatible with existing scriptpubkeys.
11:58 < morcos> gmaxwell: yes, i just got confused about whether verify shortcircuited or not
11:59 < morcos> ah so its the tx version of the spending tx that matters
11:59 < gmaxwell> As far as version, it's a little complicated. We don't have versions on txouts in transactions.  Though we do track the transaction version in the utxo set at least.
12:00 < gmaxwell> so one could use that utxo version to decide what rules to apply. alternatively one could use the spending transaction, version... and if you need to make a signature with the old rules to have it pass, then use the old version.
12:00 < gmaxwell> So I think that would work.
12:00 < morcos> yeah, but its more messy than i hope
12:00 < morcos> d
12:01 < jgarzik> one step at a time - just getting a b/g regen would be nice
12:01 < jgarzik> ;p
12:01 < gmaxwell> Not really sure if it's worth it for the existing checksig... esp since for ECDSA with our current encoding it only gives you the async gains.  For the schnorr in elements it allows a 2x verification speedup from batch verification.
12:22 < gmaxwell> sipa: can you please get cdecker to not call his work "normalized transaction ids"
12:22 < gmaxwell> (or have you met with him already?)
12:40 < sipa> gmaxwell: just did
12:40 < sipa> why not?
13:19 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has joined #bitcoin-core-dev
13:29 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev
13:36 < gmaxwell> sipa: because of the instant automatic assumption that this results in a transaction ID that cannot change.
13:36 < gmaxwell> sipa: did you see that luke figured out how to make SW a soft-fork, rather elegantly too?
13:37 < sipa> no, but i did talk about it with cdecker
13:44 < sipa> gmaxwell: well it'd be a second id... transactions would have a txid and an ntxid
13:45 < CodeShark> +1 to getting rid of scriptSig :)
13:46 < gmaxwell> sipa: but the ntxid is not actually non-malleable. So if you write software expecting that it is, you will be disappointed (and maybe even suffer funds loss).
13:47 < gmaxwell> It's non-malleable against certian kinds of operations under certian conditions, and quite useful. But we had this problem before that people assume this ID is actually more useful for accounting purposes than it is (with your prior writeup)
13:48 < sipa> anything except sighash flags that isn't covered?
13:49 < gmaxwell> signer just modifying the transaction; e.g. I pay you and it takes two hours to confirm, and during that time I RBF the payment (to boost the fees, for example).  You still get paid all the same but now your software is confused when the ntxid changed out from under you.
13:49 < CodeShark> you can also stick in some NO_OPs into the script just to mess with the system :)
13:50 < gmaxwell> And if you go look at the malleability paper from fc2015 thats the kind of thing wallets were doing with txid and getting horribly confused.
13:50 < CodeShark> or write a different script with equivalent logic
13:50 < sipa> gmaxwell: that's not malleability
13:50 < sipa> but fair enough
13:50 < sipa> do you have a better idea?
13:50 < sipa> name
13:51 < CodeShark> so the main feature is that malleating the transaction requires creating a new signature/new signatures?
13:51 < gmaxwell> I think we should do the SW instead, and then it's just the same old txid and we do not introduce yet another ID that people need to deal with which is supposted to help malleabiity but still can't actually be used as a payment identifier.
13:52 < sipa> SW doesn't help against these
13:52 < sipa> it also changes txid on resigning/modifying a transaction
13:52 < gmaxwell> Indeed, it doesn't but it doesn't introduce yet another ID which also doesn't do what people expect it to do unconditionally (because the expectation is unreasonable)
13:53 < gmaxwell> and it has a number of additional benefits. (e.g. allowing private lite wallets that don't download signatures; or syncing the history without syncing the signatures)
13:53 < sipa> i fear this is a lot more work than you think
13:53 < sipa> not just in bitcoin core
13:53 < gmaxwell> And not increasing the size of the utxo set.
13:54 < sipa> it means we need new messages for blocks and transactions
13:54 < sipa> to relay extra data
13:54 < sipa> store that on disk
13:54 < gmaxwell> You can use it in a softfork manner without updating. Actually the bitcoin protocol _used_ to relay extra data outside of transactions but that got removed.
13:54 < sipa> go fetch it from peers that have it
13:54 < gmaxwell> Yes, it complicates relay; which is the downside compared to the approach in EA.
13:55 < sipa> it introduces a new type of block witholding
13:55 < gmaxwell> sipa: I don't think it does.
13:55 < sipa> if an old client gives you a block without the witness you need to go fetch it elsewhere
13:55 < gmaxwell> sipa: or you just require that the network of updated nodes be connected.
13:56 < sipa> so you stop accepting blocks from old nodes? yuck...
13:56 < gmaxwell> E.g. you wouldn't fetch a block from a older protocol version.
13:56 < gmaxwell> which, of course, could be deployed first.
13:58 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Quit: Leaving]
13:58 < gmaxwell> In exchange you: Avoid giving the public two txids both of which don't do what they expect, you avoid increasing the UTXO set size by 30%, you allow new sync methods that don't transfer the signatures (saving 2/3rd the bandwidth), and you avoid double hashing transaction data to compute two IDs on it.
13:59 < morcos> 9 blocks in the last 20 mins?
14:00 < sipa> and you get a single txid that equally doesn't do what you expect, pretty much need everyone to upgrade anyway, or you need to keep supporting chains of transactions in which some are old-version which breaks even the intended effect
14:01 < sipa> while ntxid is strictly better at accomplishing the goal of malleability protection, and is a simple new opcode to softfork in
14:02 < sipa> (with the downsides you mentioned, yes)
14:02 < gmaxwell> sipa: the txid is functonally the same as the ntxid but at least it's not confusing people by being something new that they'll understand as existing to accomplish something that cannot be accomplished.
14:02 < gmaxwell> sipa: it is strictly inferior.
14:02 < sipa> ntxid is recursive; sw is not - if some transactions in the chain do not use sw, it breaks; ntxid does not break
14:03 < gmaxwell> sipa: okay point... This is true, but the cost is a 30% increase in utxo size. I don't think that is reasonable.
14:03 < gmaxwell> thats a significant and perminant increase in the total operating cost of the system.
14:04 < gmaxwell> as far as recursive case, it's not so simple-- for contracts all the participants are using the new style scriptpubkeys and you can enforce it so its fine.
14:05 < sipa> hmm, i thought i did the math before, and it was 20% extra
14:05 < sipa> now it seems to be 50% extra
14:05 < gmaxwell> for ntxid none of the existing software is using these IDs, so gain none of the protection, and nodes need reference rewriting to fixup transactions when their parents change.
14:06 < sipa> yeah, that fixup is annoying
14:07 < gmaxwell> I agree its somewhat less annyoing then having to provide an upgraded relay mechenism, but we need to upgrade relay regardless... and doing weird topology things to assure connectivity, but these are one time and short term costs.
14:07 < sipa> this makes no sense
14:07 < sipa> the average serialized size of txouts in the utxo set (excluding txid) is 7 bytes
14:07 < sipa>   "transactions": 7386379,
14:07 < sipa>   "txouts": 32564578,
14:07 < sipa>   "bytes_serialized": 480114635,
14:08 < gmaxwell> My 30% was just 22+22+22+4+4+32  vs that plus 32 more.
14:08 < gmaxwell> sipa: f2pools empty pubkey attack... but I didn't think it was that many.
14:09 < sipa> also, i though the total size was way larger than 480 MB
14:09 < gmaxwell> oh indeed, something there is screwed. lemme check.
14:24 < gmaxwell> Myserty solved, obfscuation change broke the stats by cherry-picking bugged code from sipa's addrindex branch.
14:24 < gmaxwell> Author: Pieter Wuille <pieter.wuille@gmail.com>
14:24 < gmaxwell> Date:   Wed Oct 7 17:12:24 2015 -0700
14:24 < gmaxwell> -                stats.nSerializedSize += 32 + slValue.size();
14:24 < gmaxwell> +                stats.nSerializedSize += 32 + pcursor->GetKeySize();
14:25 < GitHub125> [bitcoin] sipa opened pull request #6865: Fix chainstate serialized_size computation (master...fixchainsize) https://github.com/bitcoin/bitcoin/pull/6865
14:31 < sipa> so, it's a 22% increase to add ntxid at this point
14:31 < sipa>   "bytes_serialized": 1065420089
14:33 < btcdrak> sipa: that's a lot
14:34 < gmaxwell> indeed, its a figure that is artifically lowered by the spam attacks... 
14:34 -!- devrandom [~devrandom@unaffiliated/niftyzero1] has quit [Remote host closed the connection]
14:35 < sipa> lowered?
14:35 < sipa> ah, the 22% is lowered - yes
15:19 < GitHub36> [bitcoin] MarcoFalke opened pull request #6866: [trivial] fix white space in rpc help messages (master...MarcoFalke-2015-rpcWhitespace) https://github.com/bitcoin/bitcoin/pull/6866
15:50 -!- sipa [~pw@2a02:348:86:3011::1] has quit [Ping timeout: 240 seconds]
15:50 -!- sipa [~pw@2a02:348:86:3011::1] has joined #bitcoin-core-dev
15:58 -!- molly [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
16:01 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
16:22 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has joined #bitcoin-core-dev
16:22 -!- droark [~droark@c-24-22-36-12.hsd1.or.comcast.net] has quit [Client Quit]
16:46 < gmaxwell> sdaftuar: your headers first relay test... can you try that with nagle disabled? :P (e.g. is most of the benefit from the change nagle?  (trivial patch)
16:48 < sdaftuar> gmaxwell: basically all the benefit is nagle
16:48 < sdaftuar> if we turn it off, the existing relay code works as it's supposed to (one round trip instead of two)
16:49 < sdaftuar> #6494 wasn't intended to "fix" that bug, it's intended to make relaying generally work better in the case of reorgs
16:49 < sdaftuar> it just happens to not trigger nagle 
16:49 < gmaxwell> yea, obviously #6494 is good and we should do it.
16:50 < gmaxwell> oh actually it was gavin that was testing it sorry.
16:55 < GitHub20> [bitcoin] gmaxwell opened pull request #6867: Set TCP_NODELAY on P2P sockets. (master...nodelay) https://github.com/bitcoin/bitcoin/pull/6867
17:12 < Luke-Jr> re sigop-limit flooding, can we just merge acceptnonstdtxn (mainnet option, off by default) finally? this would not be a problem if people were using code with it merged..
17:15 < Luke-Jr> https://github.com/bitcoin/bitcoin/pull/559
17:16 -!- PaulCapestany [~PaulCapes@204.28.124.82] has quit [Quit: .]
17:19 < gmaxwell> Luke-Jr: explain your logic?
17:19 < Luke-Jr> gmaxwell: the top commit limits (in policy) tx sigop count based on its size
17:19 < Luke-Jr> so the spam would need to be larger for it to use 15 sigops
17:20 < Luke-Jr> thus hitting the block size limit approximately at the same time as the sigop limit
17:20 < gmaxwell> I ~think~ the suggestion I made of max(size, sigops*1e6/20000) is more general than the limit thing.
17:20 < Luke-Jr> (and triggering higher fees, of course)
17:21 < Luke-Jr> gmaxwell: yes, my point is that the bug wouldn't exist in the first place had this already been merged
17:21 -!- jl2012 [~jl2012@unaffiliated/jl2012] has quit [Ping timeout: 255 seconds]
17:23 -!- PaulCapestany [~PaulCapes@204.28.124.82] has joined #bitcoin-core-dev
17:54 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-zpbkhnefbiobmuzl] has quit [Quit: Connection closed for inactivity]
17:54 -!- PaulCapestany [~PaulCapes@204.28.124.82] has quit [Quit: .]
17:55 -!- PaulCapestany [~PaulCapes@204.28.124.82] has joined #bitcoin-core-dev
18:10 < BlueMatt> morcos: you asked about tx cache? https://github.com/TheBlueMatt/bitcoin/tree/limitucache
18:10 < BlueMatt> actually, I'll just pr
18:11 < GitHub110> [bitcoin] TheBlueMatt opened pull request #6868: Limit txn cache growth (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6868
18:13 < gmaxwell> BlueMatt: for your don't fork from here-- I suggest https://github.com/gmaxwell/secp256k1
18:13 < BlueMatt> gmaxwell: yes, but that takes effort....
18:14 < gmaxwell> well if someone does it to the repo first you can check out theirs and force push it into yours. :P
18:14 < gmaxwell> I guess I should do bitcoin core too.
18:16 < GitHub26> [bitcoin] TheBlueMatt closed pull request #6868: Limit txn cache growth (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6868
18:46 < morcos> BlueMatt: interesting idea.. i'm not sure what i think of it yet...  you don't thinks also potentially inefficient
18:48 < morcos> did you see the conversation wumpus and I were having earlier
18:48 < morcos> one thing we could do in ATMP, is not pull into the pcoinstip cache coins for txs which aren't actually accepted into the memory pool
18:56 < sipa> yeah, we should do that
18:56 < sipa> i see sometimes huge increases in utxo cache size after a transaction
18:58 < sipa> however, with bith limited utxo cache and mempool, memory usage actually remains *very* low
18:58 < sipa> have a node with 100 MB utxo and 200 MB mempool, and total memory is staying below 570 M
18:59 -!- amiller [~socrates1@unaffiliated/socrates1024] has quit [Quit: ZNC - http://znc.sourceforge.net]
19:01 < morcos> just don't run GetBlockTemplate! :)
19:02 < morcos> sipa: why does CreateNewBlock need to hold cs_main the whole time
19:02 < sipa> no idea
19:02 < sipa> also no idea why it actually needs to verify the result
19:03 < morcos> it seems like after the first round of calling HaveCoins on all the txins of all the txs.. you now have a view with all of of your coins in it
19:03 < morcos> and you could release cs_main
19:03 < sipa> the mempool could change then, though?
19:03 < morcos> i took a first stab at a background miner, but realized its useless if its holding cs_main
19:04 < morcos> well... what do you mean change?  i think we don't want to lose txs that we might be trying to mine
19:04 < morcos> but maybe you can handle that by not removing things
19:05 < morcos> a smaller lock that just guards actually removing txs
19:06 < morcos> why would it not need to verify the result, you mean because it shouldn't have been able to create something invalid
19:06 < sipa> yes
19:06 < morcos> i do think thats a nice double check 
19:06 < sipa> i don't think it's worth the cost
19:06 < sipa> given that it hasn't failed in years
19:06 < morcos> well it will be if we're chaning the mining algorithm!
19:06 < morcos> changing
19:06 < sipa> maybe a spot check would be useful
19:07 < sipa> run it once every 1000 calls
19:07 < jgarzik> since it's in the background the cost is mitigated
19:07 < morcos> holds cs_main
19:07 < sipa> block validation will not be background
19:07 < sipa> it needs the utxo set
19:08 < sipa> though i guess you are right that of it actually does make a full copy of all inlits, it can release the lock afterwards
19:08 < jgarzik> yes it needs utxo set but doesn't need cs_main long term
19:08 < sipa> well it needs the utxo lock, whatever that is
19:08 < morcos> yeah i think we could be smart about it
19:08 < sipa> i think it shouldn't need the utxo set or copies at all
19:08 < sipa> the mempool should be consistent
19:09 < sipa> do a spot check occasionally to make sure the code isn't broken
19:09 < morcos> right now it already pulls the potentially used utxos into caches TWICE.  so you could use one for creation and the other for validation without needing to grab cs_main again
19:09 < sipa> yup
19:09 < sipa> it should be 0, imho
19:09 < morcos> oh
19:09 < morcos> thats what you mean by mempool should be consistent
19:09 -!- MarcoFalke [c3523fc6@gateway/web/cgi-irc/kiwiirc.com/ip.195.82.63.198] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
19:09 < sipa> the mempool is known to not double spend
19:09 < morcos> yikes!
19:10 < sipa> not contain invalid transactions
19:10 < morcos> but thats also been broken
19:10 < sipa> so we should fix it
19:10 < morcos> but what i'm saying is that breaks often enough that you wouldn't want it to cause you to mine an invalid block?
19:10 < morcos> at least one check should still exist
19:10 < sipa> that's like saying "we're not sure about the utxo cache, let's run checkblockchain on every update"
19:10 < sipa> sure the check should exist
19:11 < sipa> but it should not be slowing down GBT by seconds!
19:11 < morcos> ok ok
19:11 < sipa> every single call
19:12 < morcos> its just scary when you have a giant block of code protected by cs_main..  trying to reason about what in there really cares about it
19:12 < sipa> well you don't want the best block or the mempool to change underneath it
19:12 < sipa> or you will be pulling in transactions that may conflict with each other
19:13 < Luke-Jr> sipa: GBT verifies the result because it's worth it. No miner wants to find out after the fact they mined an invalid block.
19:13 < morcos> if the best block changes i've got other problems..  i don't want to be calculating this block anyway
19:13 < sipa> Luke-Jr: the code should not produce invalid blocks, period
19:13 < Luke-Jr> also, spot checking just leads to people /expecting/ a lower time than they can rely on
19:13 < sipa> Luke-Jr: if you run the check once every 1000 calls you will equally well detect broken code
19:13 < Luke-Jr> not equally well, no.
19:13 < sipa> if the code is broken, it will fail badly
19:14 < Luke-Jr> and it will mislead people into thinking it's faster than it's necessarily
19:14 < Luke-Jr> an edge case will not necessarily fail badly.
19:14 < sipa> so run with -checkmempool on then
19:14 < Luke-Jr> without the verification, there is no consensus code involved in producing blocks here..
19:14 < Luke-Jr> valid transactions != valid block
19:14 < sipa> then you will also not get an invalid memlool state
19:15 < sipa> just get that check out of the mining path
19:15 -!- amiller [~socrates1@unaffiliated/socrates1024] has joined #bitcoin-core-dev
19:15 < sipa> i wonder how much gbt delay has encourage validationless mining...
19:15 < Luke-Jr> removing the check *is* validationless.
19:16 < morcos> I'm with Luke-Jr that I think consensus code should happen at least once in the path
19:16 < sipa> it is not if the code is not broken
19:16 < morcos> but it doesn't have to happen twice as it does now
19:16 < Luke-Jr> morcos: wait, twice?
19:16 < sipa> once for building, once for checking
19:16 < morcos> CheckInputs on the transactions and TestBlockValidity
19:16 -!- amiller [~socrates1@unaffiliated/socrates1024] has quit [Client Quit]
19:17 < Luke-Jr> CheckInputs doesn't check the entire block, but maybe it can be removed
19:17 < sipa> it does check the whole block eventually, as it's called for all transactions that end up in it
19:18 < Luke-Jr> sipa: no, because all the transactions can be valid without the block itself being valid
19:18 < morcos> but i think we're talking about different things here
19:18 < sipa> Luke-Jr: i know that
19:18 < morcos> i'm concerned with having the block-template thread hold cs_main too much...
19:18 < morcos> CheckInputs doesn't need cs_main once you have a view
19:18 < sipa> but checkinputs is called for the equivalent amount of work of validating the whole block
19:18 < morcos> sipa your concern seems to be latency in GBT
19:18 < morcos> but thats solvable in other ways
19:18 < sipa> yes
19:19 < sipa> ok
19:19 < morcos> as long as you think its ok to have brief periods of mining on an empty block or something
19:19 < Luke-Jr> GBT isn't supposed to be time-critical anyway (but I already plan to improve it)
19:20 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving]
19:20 < sipa> well it can't take seconds...
19:20 < Luke-Jr> maybe ideally it wouldn't, but it shouldn't hurt much
19:20 < morcos> forget about how long it takes, what you care about is how long after a new block comes in that you have a header you can mine that builds off it right?
19:21 < Luke-Jr> also afaik right now it only takes seconds when the miner is neglecting to keep a clean mempool
19:21 < morcos> it doesn't matter if it then takes seconds to generate a new one with txs right?
19:21 < Luke-Jr> since it needs to go over every tx in the mempool
19:21 < Luke-Jr> not because of the test afterward
19:21 < sipa> morcos: as long as fees are negligablez yes
19:21 < morcos> although seconds is really absurdly slow and we should be able to get it under that even for that case
19:21 < morcos> Luke-Jr: yeah thats one of the problems, is that now it loops all the txs twice
19:22 < morcos> once putting them all in vecPriority
19:22 < morcos> and then trying to go through vecPriority and put them all in a block
19:22 < Luke-Jr> yeah
19:22 < morcos> with a sorted mempool
19:22 < morcos> both become unnecessary (the 2nd was already unnecessary)
19:22 < Luke-Jr> sorted mempool is essentially what I'm working on
19:22 < Luke-Jr> although I need to rebase on top of the packages stuff I expect
19:22 < morcos> sdaftuar is very close to finishing the ancestor package tracking
19:23 < morcos> if you want to build a more general sorting priority score, thats probably what you should build off of
19:23 < Luke-Jr> not priority score
19:23 < Luke-Jr> sorting based on the user-defined policy ;)
19:23 < morcos> yeah thats what i meant
19:24 < morcos> don't you have to somehow turn their policy into a score?
19:24 < Luke-Jr> no
19:24 < morcos> how can you sort without a score?
19:24 < Luke-Jr> the policy is implemented mostly by a method that compares two transactions
19:25 -!- amiller [~socrates1@unaffiliated/socrates1024] has joined #bitcoin-core-dev
19:25 < morcos> and if  A < B and B < C then A < C right?
19:27 < Luke-Jr> right
19:27 < Luke-Jr> or at least, that's assumed by the mempool
19:27 < Luke-Jr> it might be fun to some day make blocks with a policy that just returns rand() :P
19:27 < morcos> ok well the point i was going to make was that if you could look at it as a score
19:27 < morcos> and A has child A2 and B has child B2
19:28 < sipa> you're defining a total ordering, that is equivalent to a score function
19:28 < Luke-Jr> yes, you could look at it that way; my point was that actually using a score is overcomplex
19:28 < morcos> then if you use suhas's ancestor package tracking with your score instead of fee you'll get CPFP in terms of your policy
19:31 < morcos> sipa: what are your thoughts on maintaining ancestor package tracking on nodes that aren't mining
19:31 < morcos> its not totally useless and it hopefully shouldn't be too expensive.  but certainly adds more mememory/cpu for something they are basically not using
19:32 < morcos> the one thing i think it could be used for is fee estimation?  but thats only a maybe, i haven't really figured out how or if that would be worth it.  but estimating what miners are going to do seems like potentially a valuable tool for fee estimation.
19:45 -!- CodeShark [CodeShark@cpe-76-167-237-202.san.res.rr.com] has quit []
19:45 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
19:48 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has quit [Remote host closed the connection]
19:48 -!- CodeShark [~androirc@cpe-76-167-237-202.san.res.rr.com] has joined #bitcoin-core-dev
20:01 -!- Arnavion [arnavion@unaffiliated/arnavion] has quit [Remote host closed the connection]
20:01 -!- Arnavion [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev
20:39 < gmaxwell> cfields_: good catch, apparently it's not inhereted reliably and there is a bunch of broken software out there.
20:50 < gmaxwell> (by reliably I mean it's system dependant)
21:59 -!- harding [~harding@mail.dtrt.org] has quit [Remote host closed the connection]
22:54 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
23:21 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.]
23:22 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-pkozjheyudppxmcp] has joined #bitcoin-core-dev
23:35 -!- fkhan [weechat@gateway/vpn/mullvad/x-nttkqpmmemtcgfgu] has quit [Ping timeout: 240 seconds]
23:41 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
23:47 -!- fkhan [weechat@gateway/vpn/mullvad/x-skroeccqmaqfpcro] has joined #bitcoin-core-dev