--- Day changed Tue Dec 15 2015
00:07 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
00:09 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has quit [Ping timeout: 250 seconds]
01:13 -!- raedah [~raedah@172.58.40.122] has quit [Quit: Leaving]
01:20 -!- PaulCapestany [~PaulCapes@204.28.124.82] has joined #bitcoin-core-dev
01:26 -!- ParadoxSpiral [~ParadoxSp@p508B890C.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
01:28 -!- PaulCapestany [~PaulCapes@204.28.124.82] has quit [Quit: .]
01:29 -!- PaulCapestany [~PaulCapes@204.28.124.82] has joined #bitcoin-core-dev
02:02 -!- harding [~harding@mail.dtrt.org] has quit [Remote host closed the connection]
02:24 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Quit: Leaving.]
02:47 -!- btcdrak [uid115429@gateway/web/irccloud.com/x-pqdlaxjmaavepkaj] has joined #bitcoin-core-dev
02:56 -!- MarcoFalke [8af602f0@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.240] has joined #bitcoin-core-dev
03:02 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Quit: This computer has gone to sleep]
03:06 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
03:07 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
03:20 -!- asoltys [~adam@li92-10.members.linode.com] has quit [Remote host closed the connection]
03:27 -!- laurentmt [~Thunderbi@128-79-141-196.hfc.dyn.abo.bbox.fr] has joined #bitcoin-core-dev
03:28 -!- laurentmt [~Thunderbi@128-79-141-196.hfc.dyn.abo.bbox.fr] has quit [Client Quit]
03:38 < Luke-Jr> jonasschnelli: simple_prodname updated for more testing
04:02 < jonasschnelli> Luke-Jr: okay. Thanks,... will build again.
04:09 -!- Tera2342 [~Tera2342@223.207.237.181] has quit [Ping timeout: 240 seconds]
04:13 -!- MarcoFalke [8af602f0@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.240] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
04:13 -!- MarcoFalke [8af602f0@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.240] has joined #bitcoin-core-dev
04:35 -!- afk11 [~afk11@unaffiliated/afk11] has joined #bitcoin-core-dev
04:38 -!- afk11 [~afk11@unaffiliated/afk11] has quit [Ping timeout: 240 seconds]
04:43 < phantomcircuit> wumpus, i assume CWallet::fFileBacked is used for test harness stuff?
04:48 < jonasschnelli> phantomcircuit: IMO, the fFileBackend is legacy stuff and unused.
04:51 < phantomcircuit> jonasschnelli, it's used by the recovery code apparently
04:51 < phantomcircuit> CWallet dummyWallet;
04:53 -!- Tera2342 [~Tera2342@223.207.237.181] has joined #bitcoin-core-dev
04:54 < jonasschnelli> phantomcircuit: that might be possible... right
04:57 < phantomcircuit> jonasschnelli, lol there's no good reason for it either
04:58 < phantomcircuit> and it's also used in wallet_tests.cpp
04:58 < jonasschnelli> wallet_test does use it over a file backend? not?
04:59 < jonasschnelli> test_bitcoin.cpp calls 'pwalletMain = new CWallet("wallet.dat");'
04:59 < phantomcircuit> jonasschnelli, src/wallet/wallet_tests.cpp
05:00 < jonasschnelli> Arg. Right: static CWallet wallet;
05:00 < jonasschnelli> Maybe for test reasons it could make sense... although, even there we could inject a tmp file
05:00 < jonasschnelli> (and get rid of the fFileBackend stuff)
05:17 < jonasschnelli> Luke-Jr: background has now to tiffs,... but does not appear on my mac. Maybe a DSStore problem? Why did you change the DS_Store file anyway? Because of the icon position (based on the filename)?
05:32 -!- laurentmt [~Thunderbi@128-79-141-196.hfc.dyn.abo.bbox.fr] has joined #bitcoin-core-dev
05:36 -!- laurentmt [~Thunderbi@128-79-141-196.hfc.dyn.abo.bbox.fr] has quit [Client Quit]
05:37 -!- ParadoxSpiral_ [~ParadoxSp@p508B9B18.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
05:39 -!- ParadoxSpiral [~ParadoxSp@p508B890C.dip0.t-ipconnect.de] has quit [Ping timeout: 256 seconds]
05:51 -!- Cory [~C@unaffiliated/cory] has quit [Ping timeout: 276 seconds]
06:00 -!- Quent1 [~Quent@unaffiliated/quent] has joined #bitcoin-core-dev
06:02 -!- Quent [~Quent@unaffiliated/quent] has quit [Read error: Connection reset by peer]
06:04 < GitHub138> [bitcoin] tnull opened pull request #7216: Removed offline testnet DNSSeed 'alexykot.me'. (master...delete_offline_dnsseed) https://github.com/bitcoin/bitcoin/pull/7216
06:15 -!- Guest51235 [~C@71-89-77-210.dhcp.bycy.mi.charter.com] has joined #bitcoin-core-dev
06:17 -!- Guest51235 [~C@71-89-77-210.dhcp.bycy.mi.charter.com] has quit [Client Quit]
06:18 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has joined #bitcoin-core-dev
06:20 -!- Cory [~C@unaffiliated/cory] has joined #bitcoin-core-dev
06:33 -!- challisto [~challisto@unaffiliated/challisto] has quit [Quit: Leaving]
06:43 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
06:44 -!- ParadoxSpiral_ [~ParadoxSp@p508B9B18.dip0.t-ipconnect.de] has left #bitcoin-core-dev ["cya"]
06:44 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
07:22 -!- afk11 [~afk11@unaffiliated/afk11] has joined #bitcoin-core-dev
07:35 -!- Tera2342 [~Tera2342@223.207.237.181] has quit [Ping timeout: 250 seconds]
07:37 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev
07:41 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev
07:47 -!- helo [~helo@unaffiliated/helo] has quit [Quit: leaving]
07:48 -!- helo_ is now known as helo
07:49 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has quit [Ping timeout: 250 seconds]
07:51 -!- laurentmt [~Thunderbi@128-79-141-196.hfc.dyn.abo.bbox.fr] has joined #bitcoin-core-dev
07:51 -!- laurentmt [~Thunderbi@128-79-141-196.hfc.dyn.abo.bbox.fr] has quit [Client Quit]
08:15 -!- zookolaptop [~user@c-73-217-16-2.hsd1.co.comcast.net] has joined #bitcoin-core-dev
08:45 -!- Cory [~C@unaffiliated/cory] has quit [Ping timeout: 265 seconds]
08:51 -!- Cory [~C@unaffiliated/cory] has joined #bitcoin-core-dev
09:08 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Quit: This computer has gone to sleep]
09:28 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has joined #bitcoin-core-dev
09:31 -!- Prattler [~ttttt@78-60-12-33.static.zebra.lt] has joined #bitcoin-core-dev
09:42 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has quit [Ping timeout: 246 seconds]
09:49 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has joined #bitcoin-core-dev
09:53 -!- jgarzik [~jgarzik@c-73-135-190-58.hsd1.md.comcast.net] has joined #bitcoin-core-dev
09:53 -!- jgarzik [~jgarzik@c-73-135-190-58.hsd1.md.comcast.net] has quit [Changing host]
09:53 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev
09:53 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has quit [Remote host closed the connection]
09:57 -!- grayjedi [~jamesalex@user-0c6sfdk.cable.mindspring.com] has joined #bitcoin-core-dev
09:58 -!- MarcoFalke [8af602f0@gateway/web/cgi-irc/kiwiirc.com/ip.138.246.2.240] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]
10:04 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
10:07 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has quit [Ping timeout: 255 seconds]
10:10 < sdaftuar> if anyone is up for reviewing 7062, it could use some review (one of last two PRs tagged for 0.12 that's not yet merged).
10:16 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has joined #bitcoin-core-dev
10:16 -!- paveljanik [~paveljani@79-98-72-216.sys-data.com] has quit [Changing host]
10:16 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev
10:42 -!- grayjedi [~jamesalex@user-0c6sfdk.cable.mindspring.com] has left #bitcoin-core-dev ["WeeChat 1.3"]
10:44 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has quit [Quit: desantis]
10:45 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
11:22 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has quit [Quit: desantis]
11:23 -!- desantis [~desantis@68.47.180.250] has joined #bitcoin-core-dev
11:23 -!- desantis [~desantis@68.47.180.250] has quit [Client Quit]
11:42 -!- ChainQuery-Ian [~textual@209.95.50.48] has joined #bitcoin-core-dev
11:48 -!- Quent1 is now known as Quent
11:50 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
11:51 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has quit [Client Quit]
11:51 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
11:53 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
11:58 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has left #bitcoin-core-dev []
12:01 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
12:01 < desantis> honest question: How does this channel differ from bitcoin-dev?
12:03 < sipa> This is specifically about the Bitcoin Core software, and its implementation details
12:03 < sipa> #bitcoin-dev is more for development of the protocol
12:08 -!- zookolaptop [~user@c-73-217-16-2.hsd1.co.comcast.net] has quit [Ping timeout: 256 seconds]
12:12 -!- Luke-Jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
12:12 -!- Luke-Jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-dev
12:17 < desantis> sipa: thanks!
12:23 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has quit [Quit: desantis]
12:23 < sdaftuar> why do blocks with more sigops than MAX_BLOCK_SIGOPS not get marked as failed (BLOCK_FAILED_VALID)?
12:23 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
12:24 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has quit [Client Quit]
12:24 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
12:33 < sipa> sdaftuar: thry should!
12:33 < sdaftuar> sipa: ah, good to know
12:46 < GitHub85> [bitcoin] sdaftuar opened pull request #7217: Mark blocks with too many sigops as failed (master...fix-sigops-rejection) https://github.com/bitcoin/bitcoin/pull/7217
12:57 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has quit [Quit: desantis]
12:57 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
12:58 -!- zookolaptop [~user@65.114.195.186] has joined #bitcoin-core-dev
13:04 -!- zookolaptop [~user@65.114.195.186] has quit [Ping timeout: 260 seconds]
13:11 -!- zookolaptop [~user@65.114.195.186] has joined #bitcoin-core-dev
13:12 -!- ChainQuery-Ian [~textual@209.95.50.48] has quit [Quit: Textual IRC Client: www.textualapp.com]
13:28 -!- zookolaptop [~user@65.114.195.186] has quit [Ping timeout: 256 seconds]
13:41 -!- helo [~helo@unaffiliated/helo] has quit [Quit: leaving]
13:46 -!- helo [~helo@unaffiliated/helo] has joined #bitcoin-core-dev
13:51 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
13:52 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 260 seconds]
13:53 -!- afk11 [~afk11@unaffiliated/afk11] has quit [Ping timeout: 240 seconds]
13:54 -!- raedah [~raedah@172.56.39.174] has joined #bitcoin-core-dev
13:59 -!- bitdevsnyc [bitdevsnyc@gateway/vpn/mullvad/x-lolvkyjnvmlaiupu] has joined #bitcoin-core-dev
14:12 -!- zookolaptop [~user@c-71-196-136-219.hsd1.co.comcast.net] has joined #bitcoin-core-dev
14:15 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)]
14:19 -!- treehug88 [~textual@static-108-30-103-59.nycmny.fios.verizon.net] has quit [Quit: Textual IRC Client: www.textualapp.com]
14:21 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has quit [Quit: Leaving]
14:21 -!- job_ [~job_@172.56.6.163] has joined #bitcoin-core-dev
14:22 < job_> BlueMatt gmaxwell good seeing you guys last night. a few quick questions about getblocktemplate when you've got a sec
14:22 -!- job_ is now known as jamesob
14:23 < BlueMatt> go for it
14:24 < jamesob> i) what, in particular, about getblocktemplate requires holding cs_main? is it just that CreateNewBlock needs it, or do we want a consistent snapshot of the txs to include in the template?
14:24 < sipa> yeah, consistent snapshot
14:25 < sipa> and later verification of the constructed block
14:25 < sipa> which needs access to the utxo set
14:25 < jamesob> okay, so we need a consistent view of the utxo set to validate the block template we've just constructed
14:27 < jamesob> ii) under what circumstances do we want to rebuild this blocktemplate cache we're talking about replacing the `CreateNewBlock` routine with?
14:27 < BlueMatt> jamesob: yes, CreateNewBlock needs a consistent snapshot of the utxo
14:27 < BlueMatt> so getblocktemplate shouldn't take cs_main, but CreateNewBlock will, ofc, have to
14:27 < jamesob> e.g. on new chain tip, on new txs, periodically?
14:27 < jamesob> or some/all of the above
14:27 < BlueMatt> hmm? no, keep CreateNewBlock, just cache its output in a field that getblocktemplate reads from
14:28 < BlueMatt> as for rebuilding, just call CreateNewBlock on a timer for now
14:28 < BlueMatt> we can get smarter later :)
14:28 < jamesob> right on
14:28 < jamesob> so should we start with a PR that moves CreateNewBlock to a time-rebuilt cache, then iterate from there?
14:28 < BlueMatt> and the timer can be aggressive as long as CreateNewBlock aggressively gives up cs_main and just fails when a new block is coming in
14:29 < BlueMatt> I'd assume all three things from last night can fit into one pr
14:29 < BlueMatt> I'd hope its not much code, though, again, I havent looked at it
14:29 < BlueMatt> problem with moving CNB to a background-thread by itself is you end up with way more cs_main time :(
14:29 < jamesob> it looks like there's gonna be some shuffling... might have to factor the innards of the rpc definition out into some separate bit that we can call from some scheduled routine
14:30 < jamesob> BlueMatt yeah, that's what I was thinking...
14:30 < jamesob> like, if we've got this thing acquiring cs_main on an aggressive regular schedule, that may cause MORE contention
14:30 < BlueMatt> yea, I figured half of the rpc code would have to be moved out
14:30 < BlueMatt> that wouldnt surprise me
14:31 < BlueMatt> yup, hence the requirement to drop cs_main and fail mid-process if there is "important" contention (ie a new block came in)
14:31 < BlueMatt> contention around cs_main for other things sucks, but doesnt matter too much to most users
14:31 < jamesob> ah, right. is there some existing pattern for saying "hey, this is a low-prio lock acquisition, allow interrupts"?
14:32 < BlueMatt> nope
14:32 < sipa> a large portion of the cs_main locking is actually due to ProcessMessage/SendMessage, which don't actually need cs_main (they could get their own locks that lock node-specific data)
14:32 < BlueMatt> (and dont go overengineering there, either, which would be easy to do)
14:32 < BlueMatt> ahhhhh, scope creep
14:32 < jamesob> heh
14:33 < BlueMatt> this one is wayyyy too easy to scope-creep on
14:33 < jamesob> don't worry, I'm just tryin' to hold onto my ass for now -- not going to go on any cosmic refactoring adventures ;)
14:33 < BlueMatt> adding a global boolean called fAboutToLockCSMainForBlockProcessing sucks, but.....more than that is gonna lead to bike shedding
14:35 -!- fwfwefwef [805b7112@gateway/web/freenode/ip.128.91.113.18] has joined #bitcoin-core-dev
14:35 < jamesob> so, I forget: your suggestion was to set fAboutToLock...=true once you're going to rebuild the cache, and drop out of the process if some other thread has set that flag to false, i.e. signifying that there's some more important op that needs that lock?
14:36 < BlueMatt> no, set the flag to true in ProcessMessage or AcceptNewBlock or wherever cs_main is first locked for block acceptance (from RPC or net, you really need to check both)
14:36 < BlueMatt> then CNB will poll that flag a bunch while its running, and if it sees it, bail out
14:36 -!- fwfwefwef [805b7112@gateway/web/freenode/ip.128.91.113.18] has quit [Client Quit]
14:37 < jamesob> gotcha
14:37 < BlueMatt> once you get cs_main for new block processing, you can then drop the flag and let the background CNB thread wait for its turn for cs_main
14:37 < sipa> note that such a flag technically would require its own lock :)
14:37 < BlueMatt> yup :(
14:37 < sipa> or an atomic bool (but c++11 isn't quite there yet...)
14:37 < BlueMatt> yea, this is muchhh nicer with atomics
14:37 < BlueMatt> since atomic bool is guaranteed to be lock-free
14:38 < sipa> is it?
14:38 < BlueMatt> fGonnaLockHereRealSoonNow = true; LOCK(cs_main); fGonnaLockHereRealSoonNow = false;
14:39 < BlueMatt> oops, no, sorry, atomic_flag is guaranteed, atomic_bool is not
14:39 < jamesob> cool. iii) how do we want to handle this periodic rebuild; is this going to hook into the CScheduler abstraction somehow, or would we dedicate a whole new thread for it (which I doubt)?
14:39 < sipa> use CScheduler
14:40 < jamesob> roger
14:44 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
14:46 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 260 seconds]
14:46 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has joined #bitcoin-core-dev
14:49 < jamesob> good seeing you last night as well, sipa. hope you got some sleep :)
14:50 < sipa> i did!
14:55 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev
14:56 -!- jamesob [~job_@172.56.6.163] has quit [Ping timeout: 240 seconds]
15:03 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has quit [Remote host closed the connection]
15:03 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has joined #bitcoin-core-dev
15:10 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 260 seconds]
15:12 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev
15:14 -!- bitdevsn_ [~bitdevsny@cpe-158-222-198-108.nyc.res.rr.com] has joined #bitcoin-core-dev
15:17 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has quit [Quit: desantis]
15:17 -!- bitdevsnyc [bitdevsnyc@gateway/vpn/mullvad/x-lolvkyjnvmlaiupu] has quit [Ping timeout: 255 seconds]
15:46 -!- zookolaptop [~user@c-71-196-136-219.hsd1.co.comcast.net] has quit [Ping timeout: 240 seconds]
15:47 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
16:14 -!- Tera2342 [~Tera2342@223.207.237.181] has joined #bitcoin-core-dev
16:17 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has joined #bitcoin-core-dev
16:22 -!- Tera2342 [~Tera2342@223.207.237.181] has quit [Quit: Leaving]
16:34 < morcos> jamesob: i'm also working on CNB improvements right now
16:34 < morcos> however the background threading was further down my list, so perhaps we can coordinate our work to some degree
16:35 < morcos> right now i'm refactoring CNB into a class (i should have just done that last time)
16:37 -!- Thireus [~Thireus@icy.thireus.fr] has quit [Quit: Leaving.]
16:39 -!- raedah [~raedah@172.56.39.174] has quit [Quit: Leaving]
16:41 < morcos> i'd be interested in understanding exactly what you're trying to accomplish though, b/c it's not really clear to me how to gain a whole lot without much more locking rework.
16:42 < morcos> certainly there are a couple of good ideas that should be implemented and which we've seen PR's take a stab at though:
16:42 < morcos> 1) moving TestBlockValidity to a background thread (although its not clear how easy it is to make this interuptable and this is 90% of the time of CNB right now)
16:43 < morcos> 2) returning a template based on no txs after a new tip
16:44 < sipa> making CNB as a whole run in the background would make taking TBV out of the delay very easy: just set the stored CNB result before verifying it
16:44 < morcos> 3) interrupting the cs_main lock required for block assembly (before TBV) when a new tip comes in, although now this is only 10-20ms and it really shouldn't require so much cs_main as a mempool lock which just happens to be cs_main now
16:45 < morcos> sipa: i suppose that's true.
16:46 -!- tripleslash [~triplesla@unaffiliated/imsaguy] has quit [Ping timeout: 255 seconds]
16:46 < morcos> i guess it just seems to me that its almost more logical to let the caller dictate when (how often) the new block is assembled
16:46 < morcos> the latency is irrelvant except when a new tip comes in, and thats really the interupt function you need to get right
16:47 < morcos> and then can work just as well or even better in the same thread
16:47 < morcos> longpoll returns after a) some specified amount of time has passed, and then at the end of it a new template is created or b) a new tip comes in
16:47 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 260 seconds]
16:47 < morcos> i mean it really already is a "background" thread
16:48 < morcos> you just want to return the result before running TestBlockValidity, and you want to have an interupt for new tips
16:49 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
16:50 -!- laurentmt [~Thunderbi@128-79-141-196.hfc.dyn.abo.bbox.fr] has joined #bitcoin-core-dev
16:50 < morcos> Or am I missing that there is some advantage of immediately returning a template that is Y seconds old when you call gbt instead of having to wait X seconds for a template that is X seconds old where X is by definition less than Y b/c X is the block assembly time
16:50 -!- laurentmt [~Thunderbi@128-79-141-196.hfc.dyn.abo.bbox.fr] has quit [Client Quit]
16:54 < morcos> sipa: do you have any thoughts on the feasibility of making ConnectBlock interuptable?  That seems to me its always going to be an issue with generating a lot of different blocks if you want to keep testing them.
17:04 -!- Tera2342 [~Tera2342@223.207.237.181] has joined #bitcoin-core-dev
17:23 -!- Quent [~Quent@unaffiliated/quent] has quit [Ping timeout: 265 seconds]
17:37 -!- Cory [~C@unaffiliated/cory] has quit [Ping timeout: 240 seconds]
17:41 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-dwwohvrvxfsqwidy] has quit [Quit: Connection closed for inactivity]
17:58 -!- desantis [~desantis@c-68-47-180-250.hsd1.tn.comcast.net] has left #bitcoin-core-dev []
18:10 -!- desantis [~desantis@50.153.219.10] has joined #bitcoin-core-dev
18:11 -!- desantis [~desantis@50.153.219.10] has left #bitcoin-core-dev []
18:12 -!- desantis [~desantis@50.153.219.10] has joined #bitcoin-core-dev
18:13 -!- belcher [~user@unaffiliated/belcher] has quit [Ping timeout: 246 seconds]
18:13 -!- desantis [~desantis@50.153.219.10] has left #bitcoin-core-dev []
18:14 -!- desantis [~desantis@50.153.219.10] has joined #bitcoin-core-dev
18:15 -!- desantis [~desantis@50.153.219.10] has quit [Client Quit]
18:15 -!- desantis [~desantis@50.153.219.10] has joined #bitcoin-core-dev
18:16 -!- desantis [~desantis@50.153.219.10] has quit [Client Quit]
18:16 -!- desantis [~desantis@50.153.219.10] has joined #bitcoin-core-dev
18:25 < BlueMatt> morcos: please do not further complicate getblocktemplate
18:25 < BlueMatt> we want to move complexity out of pool servers and into bitcoind, not the other way around
18:25 < BlueMatt> (pool servers are largely bit proprietary blobs that get essentially no external review, whereas bitcoin core is....not that)
18:27 < sipa> morcos: i'd rather just not always call TBV... *ducks*
18:27 < BlueMatt> sipa: noooooooooooooooooo
18:27 < BlueMatt> that shit has been so broken up until literally the latest release
18:27 < BlueMatt> there is no way in hell we should remove that check
18:27 < sipa> i am not saying remove it
18:28 < sipa> but just call it 1% or 10% of the time or so
18:28 < sipa> it's there to detect software bugs
18:28 < sipa> it's not there to prevent incorrect responses
18:28 < BlueMatt> sipa: it is absolutely there to prevent incorrect responses
18:28 < BlueMatt> up until the latest release there were known bugs which made it required
18:28 < BlueMatt> (well assuming you dont check txn as you go, which we now dont)
18:28 < sipa> ... and cause the software to crash?
18:29 < sipa> it's an assert
18:29 < BlueMatt> it isnt anymore
18:33 < BlueMatt> morcos: in any case, my view for getblocktemplate refactors, which should be a few small changes (that I suggested to jamesob last night) is
18:33 < BlueMatt> 1) Interrupt CNB when a new block comes in...you suggest block validation times are not an issue, but, really, pools do not run with our fancy hardware or mempool limiting (hell, when talking about mempool limiting we often assumed miners have a massive mempool)
18:33 < BlueMatt> 2) run CNB in the background on a loop...making getblocktemplate never block. Often when pools call getblocktemplate, they do it very often, and you'll end up blocking accepting a new block for getblocktemplate to return.
18:34 < BlueMatt> 3) return an empty block if the current block template we have is not based on the tip
18:34 < BlueMatt> 3 should be a trivial change, and 1 and 2 should be pretty easily-reviewable code
18:34 < BlueMatt> between the 3 getblocktemplate is about as optimal as we need to care, and all of the questionable logic currently in pools to emulate this can go away
18:35 -!- PRab [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has quit [Ping timeout: 240 seconds]
18:35 < sipa> 3) means that the update new tip code should also briefly lock the CNB cached value and wipe it
18:36 < BlueMatt> morcos: which looks almost identical to your proposal, but I dont think we should only be moving TBV to a background thread
18:36 < BlueMatt> all of CNB should be there, getblocktemplate should never block
18:36 < sipa> otherwise GBT needs to grab cs_main to determine whether the tip changed
18:36 < sipa> (not a problem; just realizing this)
18:36 < BlueMatt> sipa: good point
18:39 < morcos> ok hold on catching up on scrollback
18:39 -!- PRab [~chatzilla@c-68-34-102-231.hsd1.mi.comcast.net] has joined #bitcoin-core-dev
18:41 < morcos> BlueMatt: ok first of all i'm not suggesting moving more logic into pool servers.  i 100% agree that bitcoind should on its own provide near optimal mining functionality
18:41 < BlueMatt> morcos: note that I only realized how similar our proposals were after going back and reading what I wrote :)
18:41 < morcos> for your #1, what do you want to interrupt?  90% of CNB is TBV which is mostly ConnectBlock, which is why I asked sipa about interrupting it?
18:42 < morcos> I'm not necessarily opposed to moving all of CNB into a separate thread, but I have two concerns with it, both minor'ish..  
18:43 < BlueMatt> honestly I'd love to re-add the during CNB txn validity checks that were removed for 0.12, but I'll leave that one for now...in any case, in this case ConnectBlock only touches its own datastructures, so making it return when a particular flag gets set should be doable
18:43 < BlueMatt> annoying to shove that in ConnectBlock, but, doable
18:43 < BlueMatt> we could do more fucking thread::interruption_points
18:43 < morcos> a) if CNB still acquires cs_main for a lot of time, then busy running it is going to block a lot of other stuff even if somehow new tips can interrupt it, and thats not useful if the mining software isn't getting the new blocks that quickly.   you only need as many new blocks as often as they are going to switch work
18:44 < morcos> b) when you do return work to the miner, its sort of silly to return work that is old by the period of your CNB loop instead of returning it on demand
18:44 < BlueMatt> hmm? I'm confused on a) are you just talking about the other non-new-block shit that locks cs_main?
18:44 < morcos> yes
18:44 < morcos> a lot of the nodes operation does this
18:45 < morcos> acceptToMemoryPool being among the worst offenders
18:45 < BlueMatt> yea, I'm not too worried about that....mostly because the rest of that stuff is very non-latency-critical
18:45 < sipa> accepting new blocks is
18:45 < morcos> BlueMatt: as for your comment about massive mempools or fancy hardware.  CreateNewBlock speed is not related to mempool size, or only just barely.
18:46 < BlueMatt> yea, aside from a new block, nothing is latency-critical
18:46 < BlueMatt> morcos: true, I suppose that is now true...still thinking in a pre-0.12 world :)
18:46 < BlueMatt> morcos: oops, no, you misread my comment, I was suggesting accepting a new block is related to mempool size
18:46 < BlueMatt> which it still is, if you have a completely insanely large mempool
18:47 < sipa> how so?
18:47 < morcos> as for adding back in the tx validity checks.   one of the reasons i was happy to take them out is it would have been just as easy to have an error in how those were applied as have an error in mempool consistency
18:47 < BlueMatt> that plus disk i/o and cache size, and some miners have surprisingly low cache/disk throughput
18:47 < BlueMatt> morcos: in theory, sure, but we've never had a release where mempool was consistent (and we thought it was for many, many releases), but we've never found any issues in the way the txn checks were done in CNB
18:48 < morcos> in fact we do have an error in how they are applied already.  0.11 will just has happily build a block that fails TBV as 0.12 when a new soft fork activates
18:48 < BlueMatt> going back to your (b) above, meh, I'd much, much rather do that than block and make miners do some kind of crazy hacky solution because of it
18:48 < BlueMatt> morcos: oh? do we really? fuck :(
18:49 < sipa> how so?
18:49 < morcos> BlueMatt: back to b a second.  I'm confused, what are minors doing that you are calling a crazy hacky solution
18:49 < sipa> we never accept transactions that violate a future softfork into the mempool
18:49 < sipa> even a non-activated one
18:50 < BlueMatt> morcos: one major miner who happens to have a reasonable infrastructure has one pool server (with some failover stuff) controlling their hardware....connected to about 5/10 bitcoinds on the backend
18:50 < morcos> sipa: oh yeah sorry, i think i meant if you're running wiht non default policy
18:50 < BlueMatt> morcos: you have to do that for latency of accepting a new block reasons
18:50 < BlueMatt> morcos: but, really, now that I say that, it doesnt really effect what we're talking about :)
18:51 < morcos> BlueMatt: exactly.  once you do your number 3, then there is no more benefit to be gained beyond moving TBV out of thread
18:51 < BlueMatt> in any case, my thought is really that if you block gbt, pool server implementors are going to have to build smarter software that does things like threading or calling select() instead of being able to be just as effecient as the next guy by just using blocking sockets :)
18:51 < morcos> but at this point we're talking micro optimizations.  kind of depedns on how often minors want new work, and how frequently we think running CNB is reasonable.
18:52 < sipa> you'd certainly not run CNB in a busy loop
18:52 < BlueMatt> yea, we should have a discussion about how often we re-call CNB later
18:52 < sipa> that'd interfere greatly with other cs_main lockers
18:52 < BlueMatt> probably something related to how many new txn you have in mempool or so
18:52 < BlueMatt> but...first lets get the easy stuff in
18:52 < morcos> BlueMatt: thats already how it works
18:52 < BlueMatt> i thought it currently has a timeout?
18:52 < BlueMatt> meh, i havent looked at that code in too long, i guess
18:53 < morcos> BlueMatt: yeah maybe it waits the timeout, and then also requires that the # of txs changed
18:53 < morcos> but the point is it already has logic like that, tweakign the logic is easy
18:53 < BlueMatt> in any case, my thought is really that it is easier to write a pool if calls to gbt never take longer than an ms or two than if they sometimes block
18:53 < morcos> but anyway, sounds liek we agree on what we're trying to accomplish
18:53 < BlueMatt> really my goal is to make a braindead pool software just as fast as a really smart and overengineered one
18:53 < morcos> BlueMatt: ok, well if there is a difference between blocking for 10ms and blocking for 100ms then i agree with you
18:54 < morcos> you could assemble the JSON response ahead of time too
18:54 < BlueMatt> I mean fuck if I know, but if I'm writing something really braindead, if it blocks for 100ms I have to actually think about what to do, but if its 10 or 1, then I can just do whatever I think of first
18:54 < BlueMatt> heh, could do that, too
18:55 < morcos> but we should figure out how to interrupt connectblock
18:55 < BlueMatt> yea, that sounds painful
18:55 < sipa> how about we ship Varnish along with bitcoind to cache the GBT results *ducks*
18:55 < morcos> also, we should have connetblock be able to give up its cs_main after connecting inputs and before validating sigs
18:55 < BlueMatt> but we can since jamesob was asking for simple tasks, I was trying to avoid scope creep at all costs
18:56 < BlueMatt> morcos: yea, that
18:56 < sipa>     morcos: yea, that
18:58 < morcos> hmm... so for the simple first version
18:58 < morcos> 1) empty block template, all agree on that
19:00 < morcos> 2) need to not wait on TBV at least.   if we move all of CNB into its own thread...  does it just run on a schedule continuously.   is that close enough to ok...  i guess maybe so..
19:00 < BlueMatt> morcos: we used to be able to skip cs_main for the ConnectBlock call at the end to validate because we had a CCoinsViewCache that was filled with txn previns when we ran through and did the validation on each txn
19:00 < BlueMatt> now we cant so easily :(
19:00 < BlueMatt> (because ConnectBlock's calls to the CCoinsViewCache fall through to pcoinsViewTip since the cache is empty when we start checking validity)
19:00 < BlueMatt> oh, nvm
19:00 < morcos> one thing i don't like the idea of is waiting say 10 seconds to give a second template with actual txs in it after a new tip
19:01 < morcos> if you have that information at T+10ms, it seems crazy to spend 10secs or 30 or 60 building empty blocks
19:01 < BlueMatt> <BlueMatt> oh, nvm <-- nope, my previous statements was correct, ignore this comment
19:01 < morcos> we have to make sure we get that switchover right
19:02 < morcos> yeah i guess this is where i don't understand what pool server software should look like
19:03 < morcos> the longpoll idea makes sense to me... b/c bitcoind tells you when hey, here is new info you need
19:03 < BlueMatt> agreed
19:03 < morcos> vs you having to know when to ask
19:03 < morcos> so its by definition blocking for a while
19:03 < BlueMatt> I dont think most pool servers use longpolling, because it is often slower or something
19:03 < BlueMatt> i dont know why, havent looked into it
19:04 < morcos> well thats the question we need to answer
19:04 -!- Tera2342 [~Tera2342@223.207.237.181] has quit [Ping timeout: 246 seconds]
19:04 < BlueMatt> as for the switchover, another option is just to tickle the background thread when we realize the block is now out of date
19:04 < BlueMatt> lose an extra ms on switching time, but thats fine
19:04 < morcos> b/c it seems simpler to me to write braind dead pool software that long polls than pool software that has to figure out itself when it should ask bitcoind for info
19:05 < morcos> yes but how does the pool know to ask again
19:05 < morcos> is it literally going to busy call gbt and check if it changed?
19:05 < BlueMatt> some software does this, yes
19:19 -!- bitdevsn_ [~bitdevsny@cpe-158-222-198-108.nyc.res.rr.com] has quit [Ping timeout: 246 seconds]
19:22 -!- bitdevsnyc [~bitdevsny@cpe-158-222-198-108.nyc.res.rr.com] has joined #bitcoin-core-dev
19:30 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has quit [Ping timeout: 256 seconds]
19:34 -!- randy-waterhouse [~kiwigb@opentransactions/dev/randy-waterhouse] has joined #bitcoin-core-dev
19:40 -!- desantis_ [~desantis@50.153.219.10] has joined #bitcoin-core-dev
19:42 -!- desantis [~desantis@50.153.219.10] has quit [Ping timeout: 240 seconds]
19:42 -!- desantis_ is now known as desantis
19:44 -!- zookolaptop [~user@50.141.117.96] has joined #bitcoin-core-dev
19:48 -!- zookolap` [~user@2601:281:8001:26aa:7947:a45:ede9:5fd2] has joined #bitcoin-core-dev
19:49 -!- zookolaptop [~user@50.141.117.96] has quit [Ping timeout: 240 seconds]
19:50 -!- desantis_ [~desantis@50.153.219.10] has joined #bitcoin-core-dev
19:51 -!- desantis [~desantis@50.153.219.10] has quit [Ping timeout: 272 seconds]
19:51 -!- desantis_ is now known as desantis
19:56 -!- desantis [~desantis@50.153.219.10] has quit [Ping timeout: 265 seconds]
20:01 -!- bitdevsnyc [~bitdevsny@cpe-158-222-198-108.nyc.res.rr.com] has quit []
20:02 -!- desantis [~desantis@172.56.20.179] has joined #bitcoin-core-dev
20:02 -!- desantis [~desantis@172.56.20.179] has quit [Client Quit]
20:03 -!- desantis [~desantis@172.56.20.179] has joined #bitcoin-core-dev
20:15 -!- desantis [~desantis@172.56.20.179] has quit [Quit: desantis]
20:19 < btcdrak> morcos: I dont think jamesob saw your messages, says he quit before you replied.
20:21 < aj> btcdrak: sorry, i got distracted turning my htlc code into something closer to a working demo; should be up on github today or tomorrow
20:21 < btcdrak> aj: thanks!
20:21 -!- dcousens [~anon@110.22.219.15] has joined #bitcoin-core-dev
20:59 -!- Quent [~Quent@unaffiliated/quent] has joined #bitcoin-core-dev
21:14 < Luke-Jr> jonasschnelli: meh, may need to find another font still :x
21:16 < Luke-Jr> jonasschnelli: reason for generating a custom DS_Store, is that it needs the volume name in it
21:59 -!- Thireus [~Thireus@icy.thireus.fr] has joined #bitcoin-core-dev
22:09 -!- Tera2342 [~Tera2342@223.207.237.181] has joined #bitcoin-core-dev
22:24 -!- Cory [~C@unaffiliated/cory] has joined #bitcoin-core-dev
22:24 < Luke-Jr> fwiw, gentoo testing (not stable) just got GCC 5.3.0
22:48 -!- zookolap` [~user@2601:281:8001:26aa:7947:a45:ede9:5fd2] has quit [Ping timeout: 260 seconds]
22:50 < Luke-Jr> found a half-decent public domain font we can use
22:52 -!- raedah [~raedah@172.56.39.66] has joined #bitcoin-core-dev
22:58 -!- Tera2342 [~Tera2342@223.207.237.181] has quit [Ping timeout: 265 seconds]
22:59 -!- raedah [~raedah@172.56.39.66] has quit [Quit: Leaving]
23:22 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-vhxqxazqlqfxsupw] has joined #bitcoin-core-dev