--- Day changed Mon Nov 21 2016
00:00 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has quit []     
00:10 -!- aalex__ [~aalex@64.187.177.58] has joined #bitcoin-core-dev     
00:17 -!- aalex__ [~aalex@64.187.177.58] has quit [Ping timeout: 248 seconds]     
00:50 -!- achow101 [~achow101@unaffiliated/achow101] has quit [Ping timeout: 246 seconds]     
00:50 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Ping timeout: 248 seconds]     
00:52 -!- achow101 [~achow101@unaffiliated/achow101] has joined #bitcoin-core-dev     
01:01 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has joined #bitcoin-core-dev     
01:05 -!- rubensayshi [~ruben@82.201.92.138] has joined #bitcoin-core-dev     
01:16 -!- jannes [~jannes@178.132.211.90] has joined #bitcoin-core-dev     
01:22 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #bitcoin-core-dev     
01:52 < bitcoin-git> [bitcoin] laanwj pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/44adf683ad23...7490ae8b699d
01:52 < bitcoin-git> bitcoin/master 0e85204 Pieter Wuille: Add serialization for unique_ptr and shared_ptr
01:52 < bitcoin-git> bitcoin/master da60506 Pieter Wuille: Add deserializing constructors to CTransaction and CMutableTransaction
01:52 < bitcoin-git> bitcoin/master 1662b43 Pieter Wuille: Make CBlock::vtx a vector of shared_ptr<CTransaction>
01:52 < bitcoin-git> [bitcoin] laanwj closed pull request #9125: Make CBlock a vector of shared_ptr of CTransactions (master...sharedblock) https://github.com/bitcoin/bitcoin/pull/9125
01:52 < gmaxwell> \O/     
02:02 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has quit [Ping timeout: 250 seconds]     
02:03 -!- Victor_sueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev     
02:05 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Ping timeout: 260 seconds]     
02:07 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #bitcoin-core-dev     
02:07 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev     
02:12 -!- Victor_sueca is now known as Victorsueca     
02:23 < BlueMatt> sipa: did you ever benchmark how much of #9125's performance improvement was actually just the #9014 stuff? (I assume a lot, though the real benifit is compact block speedups here, which reindex cannot test)
02:23 < gribble> https://github.com/bitcoin/bitcoin/issues/9125 | Make CBlock a vector of shared_ptr of CTransactions by sipa · Pull Request #9125 · bitcoin/bitcoin · GitHub
02:23 < gribble> https://github.com/bitcoin/bitcoin/issues/9014 | Fix block-connection performance regression by TheBlueMatt · Pull Request #9014 · bitcoin/bitcoin · GitHub
02:51 < bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7490ae8b699d...c3906403c8ed
02:51 < bitcoin-git> bitcoin/master 4662553 Cory Fields: net: don't send feefilter messages before the version handshake is complete
02:51 < bitcoin-git> bitcoin/master c390640 Wladimir J. van der Laan: Merge #9117: net: don't send feefilter messages before the version handshake is complete...
02:51 < bitcoin-git> [bitcoin] laanwj closed pull request #9117: net: don't send feefilter messages before the version handshake is complete (master...feefilter-assert) https://github.com/bitcoin/bitcoin/pull/9117
02:52 -!- DigiByteDev [~JT2@101.78.224.202] has quit [Quit: DigiByteDev]     
02:52 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has joined #bitcoin-core-dev     
02:56 -!- xiangfu [~xiangfu@223.223.187.142] has quit [Ping timeout: 252 seconds]     
02:57 -!- xiangfu [~xiangfu@223.223.187.142] has joined #bitcoin-core-dev     
03:07 -!- laurentmt [~Thunderbi@80.215.234.180] has joined #bitcoin-core-dev     
03:11 -!- laurentmt [~Thunderbi@80.215.234.180] has quit [Client Quit]     
03:20 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has left #bitcoin-core-dev []     
03:48 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]     
03:48 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has joined #bitcoin-core-dev     
04:08 -!- aalex__ [~aalex@64.187.177.58] has joined #bitcoin-core-dev     
04:16 -!- aalex__ [~aalex@64.187.177.58] has quit [Ping timeout: 258 seconds]     
04:49 < jonasschnelli> wumpus: do you have plans to rebase #7729? Otherwise I can pick it up.
04:49 < gribble> https://github.com/bitcoin/bitcoin/issues/7729 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
04:49 < wumpus> yes, I'll get back to that at some point
04:50 < wumpus> happy some people are finally commenting on the API
04:52 < jonasschnelli> Yes. Thanks to last weeks meeting.
05:06 -!- cjcj [d4555899@gateway/web/freenode/ip.212.85.88.153] has quit [Quit: Page closed]     
05:20 < instagibbs> wumpus, I'm not an old timer, which is why I'm asking more basic account questions. It's always been deprecated in my Bitcoin lifetime :P
05:21 < wumpus> right, it's always been deprecated, it's time to move forward there
05:21 < wumpus> "always"
05:22 < instagibbs> But my assumption is correct, yes? That accounts gave balances which were often screwy/inconsistent, people should be doing this at higher layers by getting lists of outputs under labels or whatever?
05:23 < wumpus> yes, it's a mess, people have shot themselves in the foot using it many times, and also most expect it to be something else than it is (eg "subwallets" with their own utxos)
05:23 < wumpus> it's indeed not something that belongs at the wallet level
05:23 < instagibbs> Yes I recall finding the comment in code about using * was incorrect in retrieving same balances, but I couldn't even describe the replacement text and gave up
05:23 -!- YOU-JI [~youyouyou@y194065.dynamic.ppp.asahi-net.or.jp] has joined #bitcoin-core-dev     
05:24 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has joined #bitcoin-core-dev     
05:24 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has quit [Ping timeout: 258 seconds]     
05:29 < wumpus> so should the argument to importaddress and sendtoaddress etc be called "address" or "bitcoinaddress"? we use a mix of both, which isn't very consistent
05:30 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #bitcoin-core-dev     
05:30 < wumpus> same with importprivkey which uses "bitcoinprivkey" as argument name, but "importpubkey" which just uses "pubkey". Should make these consistent for #8811
05:30 < gribble> https://github.com/bitcoin/bitcoin/issues/8811 | rpc: Add support for JSON-RPC named arguments by laanwj · Pull Request #8811 · bitcoin/bitcoin · GitHub
05:30 < wumpus> my preference would be the shorter name, it *is* bitcoin we don't have any other context where we use address/pubkey/privkey
05:35 < instagibbs> shorter is better unless it results in overload in the context
05:36 < wumpus> right
05:36 < wumpus> changing to just "address" then
05:49 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev     
05:49 -!- BonyM1 [~BonyM-I@ua-83-227-211-4.cust.bredbandsbolaget.se] has quit [Ping timeout: 258 seconds]     
05:55 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has quit [Quit: Leaving]     
05:59 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has joined #bitcoin-core-dev     
06:04 -!- takashi [~takashi@ab169245.ppp.asahi-net.or.jp] has quit [Remote host closed the connection]     
06:05 -!- BonyM1 [~BonyM-I@ua-83-227-211-4.cust.bredbandsbolaget.se] has joined #bitcoin-core-dev     
06:09 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]     
06:09 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has joined #bitcoin-core-dev     
06:28 < bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/c3906403c8ed...b9a87b459d25
06:28 < bitcoin-git> bitcoin/master fa7cc5a MarcoFalke: Set DEFAULT_LIMITFREERELAY = 0 kB/minute
06:28 < bitcoin-git> bitcoin/master b9a87b4 Wladimir J. van der Laan: Merge #9179: Set DEFAULT_LIMITFREERELAY = 0 kB/minute...
06:28 < bitcoin-git> [bitcoin] laanwj closed pull request #9179: Set DEFAULT_LIMITFREERELAY = 0 kB/minute (master...Mf1611-blockFreeTxs) https://github.com/bitcoin/bitcoin/pull/9179
06:33 < bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/b9a87b459d25...210891143ba0
06:33 < bitcoin-git> bitcoin/master 7451cf5 jnewbery: Allow bitcoin-tx to parse partial transactions...
06:33 < bitcoin-git> bitcoin/master 2108911 Wladimir J. van der Laan: Merge #8837: allow bitcoin-tx to parse partial transactions...
06:33 < bitcoin-git> [bitcoin] laanwj closed pull request #8837: allow bitcoin-tx to parse partial transactions (master...bitcoin-tx-partial-transactions) https://github.com/bitcoin/bitcoin/pull/8837
06:41 < bitcoin-git> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/210891143ba0...0c577f2638b7
06:41 < bitcoin-git> bitcoin/master d768f15 mrbandrews: [qa] Make comptool push blocks instead of relying on inv-fetch
06:41 < bitcoin-git> bitcoin/master 3451203 Matt Corallo: [qa] Respond to getheaders and do not assume a getdata on inv
06:41 < bitcoin-git> bitcoin/master 037159c Matt Corallo: Remove block-request logic from INV message processing
06:42 < bitcoin-git> [bitcoin] laanwj closed pull request #8872: Remove block-request logic from INV message processing (master...fetchflags2) https://github.com/bitcoin/bitcoin/pull/8872
06:42 -!- BonyM1 [~BonyM-I@ua-83-227-211-4.cust.bredbandsbolaget.se] has quit [Ping timeout: 252 seconds]     
06:44 -!- crudel [crudel@gateway/shell/fnordserver.eu/x-qegvgzsnjgmsvvpn] has quit [Remote host closed the connection]     
06:47 -!- jtimon [~quassel@186.31.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev     
06:48 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 256 seconds]     
06:56 -!- BonyM1 [~BonyM-I@ua-83-227-211-4.cust.bredbandsbolaget.se] has joined #bitcoin-core-dev     
07:02 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev     
07:15 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]     
07:15 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has joined #bitcoin-core-dev     
07:18 -!- YOU-JI [~youyouyou@y194065.dynamic.ppp.asahi-net.or.jp] has quit [Quit: Leaving...]     
07:20 -!- AaronvanW [~ewout@unaffiliated/aaronvanw] has quit [Remote host closed the connection]     
07:23 -!- AaronvanW [~ewout@207pc74.sshunet.nl] has joined #bitcoin-core-dev     
07:23 -!- AaronvanW [~ewout@207pc74.sshunet.nl] has quit [Changing host]     
07:23 -!- AaronvanW [~ewout@unaffiliated/aaronvanw] has joined #bitcoin-core-dev     
07:31 -!- aalex__ [~aalex@64.187.177.58] has joined #bitcoin-core-dev     
07:34 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]     
07:34 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has joined #bitcoin-core-dev     
07:38 -!- aalex__ [~aalex@64.187.177.58] has quit [Ping timeout: 260 seconds]     
07:44 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]     
07:44 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has joined #bitcoin-core-dev     
08:03 < bitcoin-git> [bitcoin] ryanofsky opened pull request #9196: Send tip change notification from invalidateblock (master...pr/notify-tip) https://github.com/bitcoin/bitcoin/pull/9196
08:06 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev     
08:08 -!- e6xit [557335c9@gateway/web/cgi-irc/kiwiirc.com/ip.85.115.53.201] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]     
08:14 < thermoman> hi. I've got a transaction that's somehow not relayed to the network. but when doing getrawtransaction and sendrawtransaction afterwards it's relayed to the network
08:14 < thermoman> is this a known issue somehow?
08:15 < sdaftuar> thermoman: is this for a brand new transaction you created, or an old wallet transaction?
08:16 < thermoman> ok, it just was relayed to the network - but a hughe delay
08:16 < thermoman> sdaftuar: it's a brand new tx
08:16 < thermoman> we have noticed big delays since 2 or 3 days
08:16 < sipa> thermoman: define huge?
08:17 < thermoman> doing sendrawtransaction as a workaround after the initial TX the TX was then immediately visible
08:17 < sdaftuar> thermoman: can you clarify how you're telling that the tx is "visible"?  
08:17 < sdaftuar> ie, visible on some 3rd party explorer?
08:18 < sdaftuar> or are you looking at your node's relay behavior more directly?
08:18 < thermoman> lemme get that info from our dev
08:18 < thermoman> I guess 3rd party
08:18 < thermoman> like blockchain.info
08:18 < thermoman> i query the dev right now
08:20 < sipa> also, what kind of delays are we talking about? second? minutes? hours?
08:21 < thermoman> http://pastebin.com/LXT4GmnP
08:21 < thermoman> sipa, sdaftuar ^
08:22 < thermoman> delay ~20 minutes
08:24 < sipa> thermoman: can you show a few lines that come after the first broadcast?
08:30 < thermoman> sure, one moment
08:35 < thermoman> sipa: http://pastebin.com/8TEFK0Ee
08:35 < thermoman> this comes directly below it
08:35 < thermoman> the node in question is connected to a proxy node via connect=<IP>:8333
08:36 < thermoman> it seems like the connection to the proxy node got reset
08:36 < thermoman> all nodes run 0.13.1
08:36 < sdaftuar> right, so you had no peer to broadcast to initially
08:37 < sdaftuar> which would explain the delays you're seeing, though it raises the question of why you're getting disconnected
08:37 < thermoman> right. the proxy node is on the LAN
08:38 < sdaftuar> if you can share the debug log info around the time of a disconnect, that might help indicate if it's a bitcoind issue
08:38 < thermoman> you mean from the proxy node, right?
08:39 < thermoman> the proxy node isn't running with debug=1 - will have to change that
08:39 < sdaftuar> well it might be helpful to see it from both sides' point of view
08:39 < sdaftuar> if you just have one debug log, we can look at that
08:40 < thermoman> lemme activate debug=1 on the proxy node. and when the error occurs again tomorrow i'll put both debug logs on pastebin
08:40 < thermoman> ok?     
08:41 < sipa> thermoman: i believe you were not even connected to your proxy node when the transaction was creates
08:42 < sipa> oh, sdaftuar already told you thay
08:42 < sdaftuar> thermoman: does your proxy node -whitelist your internal node?
08:43 < thermoman> sdaftuar: yes
08:43 < thermoman> # Whitelist internal networks
08:44 < thermoman> whitelist=a.b.c.d/24
08:44 < thermoman> from bitcoin.conf on proxy node
08:46 < thermoman> we now have debug=1 on proxy node. we'll monitor the debug.log from the proxy for new connections from our nodes and when this happens again I will come back here
08:46 < thermoman> thanks so far
08:47 < sdaftuar> ok, sounds good.  if you have the debug.log from your internal node available where you saw the disconnect, we should be able to tell if your internal node decided to disconnect your proxy for some reason.
08:48 < sdaftuar> also, if the proxy rejected something your node sent it, causing a disconnect, we would be able to see that as well, i thnk
08:57 < thermoman> ok     
08:57 < thermoman> here is the interesting part from the node behind the proxy
09:05 < thermoman> sdaftuar, sipa: http://pastebin.com/ZzLpLiLU
09:05 < thermoman> this is the stripped down log from the node sending the TX (behind the proxy)
09:05 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]     
09:05 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-dev     
09:05 < thermoman> the issue occured several times after importing a privkey
09:06 -!- rubensayshi [~ruben@82.201.92.138] has quit [Remote host closed the connection]     
09:07 < sdaftuar> ah.  i wonder if your node was too busy to respond to ping messages, causing a disconnect?
09:07 < thermoman> that might be
09:07 < thermoman> it seems the import did block everything else
09:07 < thermoman> and directly after the import the TX was created (while being disconnected from the proxy)
09:08 < thermoman> should timeouts be upped on our nodes?
09:08 < sdaftuar> the second disconnect seems a bit different though
09:08 < thermoman> yeah
09:09 < thermoman> it's directly after
09:09 < thermoman> Erased 100 orphan tx from peer 1
09:09 < thermoman> should the nodes behind the proxy whitelist the proxy-ip, too?
09:09 < thermoman> at the moment only the proxy has whitelisted the client ips
09:09 -!- grubles [~grubles@unaffiliated/grubles] has joined #bitcoin-core-dev     
09:10 < sdaftuar> thermoman: i don't think it would hurt, though i'm not sure yet what's going on
09:14 < sdaftuar> ok i guess the interesting thing about the second disconnect is that your node tried to connect to it at 15:28:07, and then didn't send the version message untl 15:51
09:14 < sdaftuar> so this looks like the wallet operation is contending with the p2p code, and network timeouts are being hit
09:15 < thermoman> ah ok, didn't miss the connection between "Added connection" and "sending version message"
09:15 < thermoman> s/didn't/did/
09:16 < thermoman> current workaround for us is to wait a minute after importing is done
09:17 < sdaftuar> thermoman: seems like a reasonable workaround for now.  thanks for reporting this, i think this is something we should look into and improve
09:22 < thermoman> do you add an issue in github or should I?
09:23 < thermoman> we've only seen this issue after upgrading the proxy node von 0.11.2 to 0.13.1
09:24 < thermoman> the node with the TX was running 0.11.2 until some hours ago and we noticed this behaviour with 0.11.2 and even after upgrading the node to 0.13.1
09:24 < thermoman> so it seems to have started after upgrading the proxy node
09:25 < thermoman> if I don't hear from you I'll open an issue on github tomorrow
09:25 < sdaftuar> hmm, this issue really seems like it's specific to the internal node.
09:25 < sdaftuar> please go ahead and open the issue, thanks!
09:25 < thermoman> we never had this with 0.11.2 on both nodes
09:25 < thermoman> (or didn't notice?)
09:26 < sdaftuar> my memory is failing me right now about when and whether we've made changes from 0.11 to 0.13 with locking/thread contention.  so in my view it's possible there's a performance regression, i'd need to investigate more.
09:26 < thermoman> ok, thanks so far. will open the issue tomorrow
09:43 < sdaftuar> thermoman: do you often call importprivkey in your setup, ie you were doing that on 0.11.2 as well as regularly on your 0.13.1 nodes?
09:44 < sdaftuar> it looks to me like that is always a slow operation, and will cause bitcoind to be unresponsive while rescanning
09:47 < sdaftuar> (and i think this would have been true in 0.11.2 as well)
09:49 -!- crudel [crudel@gateway/shell/fnordserver.eu/x-oounzpacvdrrsscu] has joined #bitcoin-core-dev     
09:52 -!- paveljanik [~paveljani@79.98.72.176] has joined #bitcoin-core-dev     
09:52 -!- paveljanik [~paveljani@79.98.72.176] has quit [Changing host]     
09:52 -!- paveljanik [~paveljani@unaffiliated/paveljanik] has joined #bitcoin-core-dev     
09:52 -!- AtashiCon [arnavion@unaffiliated/arnavion] has quit [Quit: AtashiCon]     
09:57 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 265 seconds]     
10:04 < gmaxwell> sdaftuar: importing a private key with rescan caused timeouts and disconnects for me w/ 0.12 too. If thermoman saw a change it could have also just been a bit of chain growth taking it above threshold.
10:09 -!- AtashiCon [arnavion@unaffiliated/arnavion] has joined #bitcoin-core-dev     
10:14 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev     
10:17 -!- laurentmt [~Thunderbi@80.215.234.180] has joined #bitcoin-core-dev     
10:18 -!- laurentmt [~Thunderbi@80.215.234.180] has quit [Client Quit]     
10:23 < sipa> gmaxwell: or rescans became a bit slower since
10:38 -!- cysm [cysm@unaffiliated/paracyst] has joined #bitcoin-core-dev     
10:47 -!- To7 [~theo@cpe-158-222-222-232.nyc.res.rr.com] has quit [Quit: Whatever]     
11:04 < sdaftuar> instagibbs: in #9194, wouldn't it make sense (if we're going down this road) to also support legacy serializations for zmq and rest?
11:04 < gribble> https://github.com/bitcoin/bitcoin/issues/9194 | Add flags to getrawtransaction and getblock to return non-segwit seri… by instagibbs · Pull Request #9194 · bitcoin/bitcoin · GitHub
11:10 < instagibbs> sdaftuar, yes, I have literally no experience with those interfaces, what would that entail
11:13 < instagibbs> fwiw I think the number of spots we'll need this for is basically two, the ones I've done. We support backwards compatbility when talking to non-segwit nodes(and maybe future serializations!), I'm just giving an option for that at rpc level.
11:13 < instagibbs> so whatever makes sense for that
11:28 < sdaftuar> instagibbs: see for instance rest.cpp:371
11:29 < sdaftuar> oh wait, i made a mistake
11:31 < sdaftuar> oh maybe not -- sorry i've never really looked at this code. looks like that function gets a transaction and serializes it for a user
11:32 < sdaftuar> similarly in the rest_block() function in that same file
11:32 < sdaftuar> and then i think there are two analogous places in zmq/zmqpublishnotifier.cpp
11:33 < sdaftuar> maybe that's it though... if your changes were comprehensive for the RPC stuff (sorry I need to review still), then perhaps this isn't as bad as I feared
11:53 -!- dermoth [~thomas@dsl-66-36-147-38.mtl.aei.ca] has quit [Quit: Ex-Chat]     
11:54 -!- asoltys [~bitcoinco@23.94.96.232] has quit [Remote host closed the connection]     
11:58 < instagibbs> gettransaction also returns a hex field I'm not converting, but it feels weird messing with an api that is wallet-based for this purpose.
11:59 < instagibbs> I think "raw" apis are a better fit for these changes. If you're using wallet features, just use wallet and what the wallet understands
12:02 < gmaxwell> the overall goal should be "no one should need to hesitate to install 0.13.1+ simply because they haven't updated their applications yet".  This was part of the attractiveness of having the segwit code in a release ahead of activation and avoiding behavior changes at activation time.
12:03 -!- cysm [cysm@unaffiliated/paracyst] has quit [Ping timeout: 245 seconds]     
12:25 -!- cysm [cysm@unaffiliated/paracyst] has joined #bitcoin-core-dev     
12:34 < kanzure> should there be a nonce in rpc client requests? otherwise how does sendmany work if you had a socket timeout on the response? or other must-only-run-once critical rcp endpoints.
12:35 < kanzure> 'scuse me, *rpc endpoints
12:36 < gmaxwell> kanzure: 'check listtransactions' :-/  you're right, it should have a nonce. Well the whole rpc thing was not well designed.
12:36 < kanzure> alternatively, you could say "well, just call an event log, and check before you retry" (someone posted a "retry all rpc requests" to petertodd's python-bitcoinlib repo and it raised my eyebrow)
12:37 < kanzure> if sending money is the only scenario where something bad happens, then listtransactions is probably sufficient...
12:37 < gmaxwell> (or, better, a nonce, sequence number pair, and a nonce can be reused but only with a strictly greater sequence number)
12:39 < kanzure> what about: in RPC request to sendmany (or whatever else is broken here), include expected state and also the actual request. and if the expected state doesn't match, then raise error or don't send.
12:42 < kanzure> someone outside the channel proposes, 'if you want to enforce balance checking, make the server require the client to make a balance call before making a send call.'
12:51 < kanzure> do these problems exist in the http api?
12:56 < sipa> you mean the REST api? that's read only
12:56 < sipa> and only for public data (nothing wallet related)
13:03 < sipa> gcc 7.0 has -Wmisleading-indentation 
13:04 -!- sanada [sanada@36-2-119-80.chiba.ap.gmo-isp.jp] has quit [Ping timeout: 240 seconds]     
13:09 -!- Alina-malina [~Alina-mal@unaffiliated/alina-malina] has quit [Ping timeout: 250 seconds]     
13:11 -!- sanada [~bitktn@36-2-119-80.chiba.ap.gmo-isp.jp] has joined #bitcoin-core-dev     
13:11 -!- jannes [~jannes@178.132.211.90] has quit [Quit: Leaving]     
13:12 -!- Alina-malina [~Alina-mal@37.157.216.188] has joined #bitcoin-core-dev     
13:30 -!- cysm [cysm@unaffiliated/paracyst] has quit [Ping timeout: 258 seconds]     
13:32 < Chris_Stewart_5> Just to be clear, there is no compact size uint to indicate how many CTxInWitness inside of the serialization of a tx right?
13:32 < Chris_Stewart_5> You need to infer it from the number of inputs?
13:39 -!- ryan`c is now known as ryan-c     
13:43 -!- CubicEarth [~cubiceart@2002:329f:7e15:0:6c2a:c4d5:dc8b:8cc6] has joined #bitcoin-core-dev     
13:47 < sipa> Chris_Stewart_5: indeed
13:49 < kanzure> okay i posted an issue about the abovementioned rpc replay protection concerns https://github.com/bitcoin/bitcoin/issues/9197
13:56 < Chris_Stewart_5> Thanks sipa 
14:05 < bsm1175321> Are there any RPC requests that would NOT be idempotent?  I think kanzure's suggestion is that bitcoind always complete the action of an RPC call even in event of an I/O error while sending the response.  Is it possible that this wouldn't be the case?
14:05 < bsm1175321> Further, is it possible for a truncated RPC request to end up being valid and acted upon?
14:06 < bsm1175321> BTW I think this is probably RPC "atomicity" not "idempotency".
14:11 < kanzure> no, my suggestion is not "always complete the action of an RPC call even in event of an I/O error while sending the response" -- that's a related topic i guess.
14:11 < bsm1175321> Nah kanzure's right in using idempotent -- CS lingo.  Too much math over here.
14:12 < kanzure> my suggestion is "change from 'always execute any RPC request' to 'check whether the nonce is the same first, and fail if the nonce is the same'"
14:13 < bsm1175321> Sure.  From the client's perspective the two situations are the same...he didn't get the response and doesn't know if the RPC sendmany was executed.  So he retries with the same nonce...
14:14 < kanzure> yes, retrying with the same nonce would be OK
14:14 < sipa> an alternative is having meta support for any asynchronous command
14:14 < bsm1175321> I like that idea...
14:14 < bsm1175321> (I like kanzure's nonce idea).  What's meta support?
14:14 < sipa> you call an RPC "schedule" with 3 args: 1) command 2) command args 3) call id
14:15 < sipa> and then you have a separate RPC through which you can chrck the result
14:15 < kanzure> have you poked at database internals before, like transactional concept?
14:15 < sipa> define 'poked'
14:16 < gmaxwell> sipa: that kind of framework is _necessary_ for some commands.
14:16 < kanzure> eh, 'poked' could be reading i guess :)
14:16 < gmaxwell> For example automatic sendmany batching and replacement will not know the txid until later. 
14:17 < kanzure> daisychaining and 'schedule' sure. although if you require some of the prior information before generating the next call, ... can't send all at once.
14:18 < bsm1175321> To be clear -- the issue we're having is that bitcoind closes its RPC connections on a timer.  These are connections on localhost and python-bitcoinlib doesn't reopen them (I consider that it's bug).  But this could also be improved on the bitcoind side if we could configure it to not close RPC connections...
14:18 -!- skyraider [uid41097@gateway/web/irccloud.com/x-vlufmivotbgisqqx] has joined #bitcoin-core-dev     
14:18 < bsm1175321> That's separate from calling RPC's over a flaky connection...
14:18 < kanzure> rpc connection closing behavior is somewhat unrelated; your application still needs to know whether to retry. and currently it can't possibly know that.
14:18 < bsm1175321> Yep yep
14:18 < gmaxwell> "not close RPC connections." < guarentees file descriptor exhaustion.  And is unlike any http server ever.
14:19 < gmaxwell> bsm1175321: I think your concern is just a "fix the client" concern. (the python-bitcoinlib behavior has burned me many times too)
14:19 < bsm1175321> But there's only one RPC user and it's on localhost.  I think this is a common configuration.
14:20 < bsm1175321> gmaxwell: I agree.  I'm gonna fix it to re-open and not bother me with I/O exceptions.
14:20 < kanzure> "only one rpc user" what?
14:20 < sipa> didn't we fix that issue in our in-tree fork of python-bitcoinlib?
14:20 < gmaxwell> often still results in exhaustion, e.g. when you get long idle connection objects that aren't GCed yet and so they've left their connections open.
14:21 -!- CubicEarth [~cubiceart@2002:329f:7e15:0:6c2a:c4d5:dc8b:8cc6] has quit [Read error: Connection timed out]     
14:22 < bsm1175321> sipa: yes it looks like it is fixed there.
14:22 -!- cysm [cysm@unaffiliated/paracyst] has joined #bitcoin-core-dev     
14:22 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has joined #bitcoin-core-dev     
14:24 < kanzure> oh there is an "id" parameter sent https://github.com/petertodd/python-bitcoinlib/blob/2e4d51a482a4331d91de9bae2ab7be56229e664a/bitcoin/rpc.py#L188
14:24 < kanzure> .. which is auto-incremented in this particular client library..
14:26 < bsm1175321> sipa: the fix here: https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/test_framework/authproxy.py#L134 looks like it is effectively a retry, and would, e.g. sendmany again.
14:29 < skyraider> connection handling is distinct from client state management. i think kanzure's concern is that it seems possible for a bitcoind client to say "hey, i didn't receive a reply" and then resend the same request to bitcoind - perhaps resulting in the accidental execution of two identical requests. banks used to behave this way when you clicked "do wire transfer"
14:29 < skyraider> then pressed "back" in internet explorer - they'd do another POST to send the wire transfer again. now we have idempotency via request nonces for the web, so it's not a problem - your nonce is considered expired once the nonce's corresponding request is processed. with python-bitcoinlib in particular, i think the concern is that you can still get "oops i
14:29 < skyraider> sent wire transfer twice" type behavior, due to no client state management facilities in bitcoind to code against.. is that right?
14:31 < gmaxwell> skyraider: applications can check list transactions to determine if the order has been executed yet.
14:31 < sipa> kanzure: that's just for ordering of responses within one connection
14:32 < gmaxwell> A common model is to poll outstanding requests, check list transactions, anything not paid yet, pay... anything already paid remove from the list.
14:32 < kanzure> gmaxwell: no, that doesn't work.   listtransactions will  not matter because you might have anohter concurrent attempt. then both sendmanys execute one after another.
14:33 < kanzure> ok well at the very least we should insist that client librares implement that pattern, maybe by rejecting sendmany rpc requests if previous required rpc requests have not been detected
14:33 < gmaxwell> "Don't do that."
14:33 < gmaxwell> I'm just describing to you an architecture I've seen before, which has generally acceptable properties.
14:34 < gmaxwell> I don't disagree that an optional nonce/sequence with replay protection would be good-- though it has some scalability challenges. 
14:34 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Quit: :)]     
14:35 < kanzure> sipa: oh like for ordering responses to the _batch call?
14:36 < sipa> kanzure: you can send two rpcs with the same id, and you could get two different answers with the same id back
14:37 < sipa> gmaxwell, kanzure: a more lightway idea, just for send* calls, is passing as an argument to the rpc the expected wallet balance
14:37 < sipa> anything that actually interferes would cause failure, while other rpcs could still happen concurrently
14:37 < kanzure> wallet balance isn't good either... many simultaneous deposits/debits could put you back to the same balance anyway.
14:38 < kanzure> excuse me, credits/debits
14:38 < skyraider> so 'read before you try to send lots of bitcoin' - alright. but if another RPC connection writes in the meantime, won't my read be stale, or is there transaction isolation somehow? is the recommended policy therefore 'only ever use a single RPC writer' ?
14:38 < gmaxwell> sipa: ugh. so if you send some and recieve the same amount ... the double payment still goes through?
14:39 < kanzure> 'transaction isolation' probably means database-kind of transaction model, not transactions?
14:39 < skyraider> uh, yeah, not bitcoin transaction. isolation as in the I from ACID.
14:39 < gmaxwell> skyraider: yes, you just retry in that model.
14:40 < kanzure> isn't that a failure
14:40 < kanzure> the idea is to prevent stales from causing you to retry
14:41 < gmaxwell> why would that be a failure? The retry causes no harm. The system is concurrent and you want an atomic change, there will always be retries needed.
14:41 < kanzure> you mean existing system, or proposed system is concurrent?
14:41 < gmaxwell> kanzure: so going back to the nonce, how do you propose avoding having to store nonces forever?
14:42 < skyraider> gmaxwell: sorry, could you clarify please - is RPC safe for concurrent writes at the moment, or are you saying stale reads (if you have, say, two concurrent connections) are possible, and therefore one should use a single RPC writer?
14:43 < gmaxwell> skyraider: your question is underspecified. It's perfectly fine to make concurrent writes.  But kanzure is asking about what happens when you get a socket failure on one of your writes. 
14:43 < gmaxwell> If you just retry you may double pay.
14:43 < midnightmagic> wtf man another earthquake near fukushima?
14:47 < skyraider> right, i don't mean to ask whether the application will crash if you do concurrent writes. rather, whether a client can expect the situation of "1) read balance in client thread A, 2) write balance in client thread B that makes 1's read stale, 3) write balance in client thread A" can result in the client double paying, due to making a decision based upon the
14:47 < skyraider> balance data in read 1).
14:48 < sipa> skyraider: there is no 'write balance'
14:49 < sipa> there is 'attempt to make a transaction'
14:49 < skyraider> yes, sorry, shorthand for "try to send some bitcoin"
14:51 < gmaxwell> skyraider: I think you're overthinking things now.  If you call send* you will probably send.  If you call send once per payment you want to make you will never double pay.
14:51 < gmaxwell> if you are doing things based on 'balance' then god knows what will happen, as balance is not monotone. :)
14:51 < sipa> skyraider: if you "attempt to make a transaction" twice, it will obviously send twice... anything else would be highly unexpected behaviour
14:52 < kanzure> i think bitcoind seems to really very strongly want all applications to have only one rpc connection
14:52 < gmaxwell> kanzure: not at all.
14:52 < kanzure> there is no protection against stale reads at all
14:52 < kanzure> there is nothing in here at all like ACID
14:53 < gmaxwell> which is orthorgonal to "have only one rpc connection"
14:53 < kanzure> look i'm fine with one rpc client in serial. that's fine with me.  but i'm not sure everyone else knows to do that.
14:57 < skyraider> it might help to put on a client's shoes here. context was https://github.com/petertodd/python-bitcoinlib/pull/128 in which a blind retry python decorator was considered. turns out this can a) result in actually sending bitcoin multiple times, b) result in decisions made on stale read data (balance was just an example - really any bitcoind state subject to
14:57 < skyraider> change via concurrent writes). it seems a safe solution for b) is "only ever execute RPC writes serially" and for a), "because we are doing serial writes, in which another write can't occur after my safety-check-read, we can query to see whether the send already occurred (the safety-check-read) before retrying." tldr; serial writes enables "should i retry?"
14:57 < skyraider> logic in a safe way.
14:57 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has joined #bitcoin-core-dev     
14:58 < gmaxwell> I disagree.
14:58 < gmaxwell> What you want is a write which is ordered with respect to a read.
14:58 < gmaxwell> Have you already paid is not something which can become untrue.
14:58 < kanzure> previous answer to "have you already paid" can become untrue
14:59 < gmaxwell> yes but only by performing a write that makes that payment.
14:59 < skyraider> yes, a write ordered with respect to a particular read (i.e., transaction isolation in the database sense) is a more general and better solution than serial writes.
15:01 < gmaxwell> Both of you keep repeating serial, and that is just wrong and in a meaningful way compared to typical usage.  You can perform as many sends as you want. But if you want to retry a send you must read first to make sure the send wasn't already performed.   Running two sends for the same payment in parallel is insanity in any case, as you have no way of knowing if the other went through (and usuall
15:01 < gmaxwell> y it does).
15:01 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has quit [Remote host closed the connection]     
15:04 < bsm1175321> Of course, there are probably many people performing this logic now, since the "commit" of a send doesn't actually occur until it's mined.  The need to fee-bump is probably a more common cause of retries than RPC connection drops...
15:04 < skyraider> if you are e.g. running a group of ec2 servers or something and experience a temporary DNS failure of the kind that's common on AWS, that python-bitcoinlib decorator could result in probably 1-50 double payments per month, assuming you are sending every 30 seconds or so. serial writes seems to be a quick, safe way to get don't-pay-twice safety while querying
15:04 < skyraider> the current version of bitcoind. as to running sends concurrently, yes, it seems unusual / nonrealistic use case to do this.
15:05 < kanzure> fwiw i don't actually use sendmany
15:07 < kanzure> or any spends for that matter... heh.
15:07 < sipa> you hodler
15:08 -!- CubicEarth [~cubiceart@c-50-159-126-21.hsd1.wa.comcast.net] has joined #bitcoin-core-dev     
15:08 < kanzure> (yeah who would ever want to spend coins??)
15:10 < midnightmagic> gmaxwell: that's how MP (operating manually) managed to rip himself off.
15:13 < kanzure> midnightmagic: go on?
15:13 < skyraider> and yeah i understand the criticism about serial writes - it seems at first it should be more like 'read before you write'.  but, you have to be careful with that because your read could easily be stale. and if you are running concurrent bitcoind sends in different client-application-level thread of execution, well - you shouldn't do this. you should write
15:13 < skyraider> in one thread because the lack of transaction isolation (read/write ordering) means client implementors might not know to handle what happens when a balance they just checked is now gone.
15:13 < skyraider> but, yes, this is not an actual real-world use case over here at the moment. so admitted.
15:14 < skyraider> thanks
15:14 < midnightmagic> kanzure: not respending himself, instead sending multiple sends to the same destination more than once, and the receivers obviously refused to give anything back (which was fair by MP's own mercenrary rules.) If he had properly respent himself, updated his sends, or otherwise presumed the sends could be mined at any time, he wouldn't have lost anything.
15:22 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev     
15:23 < kanzure> is the assumption that there is always a single node worker and never more than one? if so, we should probably add this to docs.
15:23 < kanzure> *node worker (rpc client communicating to bitcoind)
15:25 < bsm1175321> Having more than one violates the Isolation requirement of ACID.  Would it be interesting to require isolation on open wallets WRT rpc clients?  That is, two RPC connections must have mutually exclusive wallets?
15:28 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Quit: laurentmt]     
15:30 < gmaxwell> kanzure: I keep telling you that there is no such assumption.
15:33 < sipa> kanzure: that also doesn't in any way help with the double issue problem
15:53 < kanzure> ah, there is a lockunspent at least.
16:18 -!- wasi [~wasi@gateway/tor-sasl/wasi] has quit [Ping timeout: 245 seconds]     
16:27 -!- JackH [~laptop@79-73-184-38.dynamic.dsl.as9105.com] has quit [Remote host closed the connection]     
17:14 -!- mol is now known as moli     
17:20 -!- go1111111 [~go1111111@104.200.154.82] has quit [Quit: Leaving]     
17:32 -!- windsok [~windsok@45.63.59.8] has quit [Ping timeout: 260 seconds]     
17:33 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
17:42 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-rabvyppajkfqhwto] has quit [Quit: Connection closed for inactivity]     
17:43 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds]     
17:45 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has joined #bitcoin-core-dev     
17:51 -!- skyraider [uid41097@gateway/web/irccloud.com/x-vlufmivotbgisqqx] has quit [Quit: Connection closed for inactivity]     
17:56 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
18:00 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev     
18:16 -!- windsok [~windsok@45.63.59.8] has joined #bitcoin-core-dev     
18:43 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has joined #bitcoin-core-dev     
18:44 -!- Cory [~Cory@unaffiliated/cory] has quit [Ping timeout: 240 seconds]     
18:47 -!- Pasha [~Cory@unaffiliated/cory] has joined #bitcoin-core-dev     
18:54 -!- Pasha is now known as Cory     
18:55 < bitcoin-git> [bitcoin] gmaxwell opened pull request #9199: Always drop the least preferred HB peer when adding a new one. (master...remove_high_bandwidth_zombies) https://github.com/bitcoin/bitcoin/pull/9199
18:58 -!- abpa [~abpa@96-82-80-25-static.hfc.comcastbusiness.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
19:04 -!- vanishing-coins [369924af@gateway/web/freenode/ip.54.153.36.175] has joined #bitcoin-core-dev     
19:04 < vanishing-coins> hey guys, I believe I have found a rather serious issue with 0.13.1 but I am trying to figure out what is causing it
19:04 -!- brg444 [415ce2de@gateway/web/freenode/ip.65.92.226.222] has joined #bitcoin-core-dev     
19:04 < vanishing-coins> the symptom is that as coins are being sent through standard sendtoaddress RPC calls, overtime some change gets 'lost' until the wallet trends to 0, and the true balance is only restored when restarting Bitcoind. I am not speaking hyperbolically
19:04 < vanishing-coins> coins are never 'lost' but bitcoind has to restart for the correct balance to be shown
19:05 < vanishing-coins> I help multiple clients manage their nodes and this is the second time that I have seen this happen 
19:05 < vanishing-coins> both clients were running 0.12 before without issue, upgraded to 0.13.1 and HD wallets and both had the issue within 48 hours of upgrading
19:08 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-obvnabvxecgawgrd] has quit [Quit: Connection closed for inactivity]     
19:08 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has quit [Quit: Leaving.]     
19:15 < vanishing-coins> both are running on m1.smalls on Amazon EC2
19:15 < vanishing-coins> 1.7 gigs of ram, is it possible that unexpected behavior manifest due to low RAM? what would I be grepping for in the debug.log if so
19:17 < vanishing-coins> the max memory pool is set to 300mb
19:23 -!- jtimon [~quassel@186.31.134.37.dynamic.jazztel.es] has quit [Ping timeout: 252 seconds]     
19:29 < instagibbs> vanishing-coins, do you have debug.log to share?
19:30 < instagibbs> it sounds awfully similar to if you somehow chain too many transactions in mempool, it can throw an error and "lose" coins until restart
19:32 < vanishing-coins> instagibbs i do 
19:32 < vanishing-coins> and that is definitely very possible -- can you elaborate on that chaining issue?
19:32 < vanishing-coins> is that a product of having too small a memory pool? any link i can read up on?
19:32 < instagibbs> well a look at the debug log when this happens would be helpful
19:33 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds]     
19:33 < instagibbs> no, sorry it's likely not the issue, just thinking out loud. debug.log sharing is best
19:33 -!- jyap [~jyap@unaffiliated/jyap] has quit [Ping timeout: 245 seconds]     
19:39 -!- jyap [~jyap@2604:180:1:7f5::b59a] has joined #bitcoin-core-dev     
19:39 -!- jyap [~jyap@2604:180:1:7f5::b59a] has quit [Changing host]     
19:39 -!- jyap [~jyap@unaffiliated/jyap] has joined #bitcoin-core-dev     
19:44 -!- DigiByteDev [~JT2@n218250011174.netvigator.com] has joined #bitcoin-core-dev     
19:44 -!- jtimon [~quassel@186.31.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev     
19:45 -!- vanishing-coins [369924af@gateway/web/freenode/ip.54.153.36.175] has quit [Ping timeout: 260 seconds]     
19:46 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has quit [Quit: WeeChat 1.5]     
19:47 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #bitcoin-core-dev     
19:55 -!- jtimon [~quassel@186.31.134.37.dynamic.jazztel.es] has quit [Ping timeout: 260 seconds]     
19:56 < gmaxwell> instagibbs: that wouldn't be until restart, no, it would be until after some of the chain confirms and then the retransmit logic fires.
20:02 -!- DigiByteDev [~JT2@n218250011174.netvigator.com] has quit [Quit: DigiByteDev]     
20:08 -!- vanishing-coins [369924af@gateway/web/freenode/ip.54.153.36.175] has joined #bitcoin-core-dev     
20:08 < vanishing-coins> hey sorry got disconnected
20:09 < vanishing-coins> I am uploading the debug.log now
20:10 < vanishing-coins> I also dropped memorypool size to 150 mb, and dbcache to 80
20:10 < vanishing-coins> but very strange all around that two completely separate instances on EC2 of the same type were running fine on 0.12 but both, within 72 hours of upgrading, had the same exact issue :/
20:23 -!- vanishing-coins [369924af@gateway/web/freenode/ip.54.153.36.175] has quit [Ping timeout: 260 seconds]     
20:26 -!- brg444 [415ce2de@gateway/web/freenode/ip.65.92.226.222] has quit [Ping timeout: 260 seconds]     
20:43 -!- abpa [~abpa@2602:306:b837:dbf0:10e0:d560:7f6f:8a8] has joined #bitcoin-core-dev     
20:46 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev     
20:47 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev     
20:49 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 258 seconds]     
20:50 -!- DigiByteDev [~JT2@n218250011174.netvigator.com] has joined #bitcoin-core-dev     
20:50 -!- DigiByteDev [~JT2@n218250011174.netvigator.com] has quit [Client Quit]     
20:51 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 260 seconds]     
21:10 -!- Cory [~Cory@unaffiliated/cory] has quit [Ping timeout: 258 seconds]     
21:11 -!- Pasha [~Cory@unaffiliated/cory] has joined #bitcoin-core-dev     
21:13 -!- DigiByteDev [~JT2@n218250011174.netvigator.com] has joined #bitcoin-core-dev     
21:18 -!- Pasha is now known as Cory     
21:53 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]     
21:54 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev     
22:03 -!- fanquake [~fanquake@unaffiliated/fanquake] has joined #bitcoin-core-dev     
22:21 < luke-jr> cfields: ping
22:21 < luke-jr> should we be doing UpdateUncommittedBlockStructures in GBT proposals? :x
22:22 < cfields> luke-jr: are gbt proposals used?
22:23 < luke-jr> cfields: that's currently how I'm trying to test :D
22:23 < cfields> luke-jr: heh, i've been using #9000
22:23 < gribble> https://github.com/bitcoin/bitcoin/issues/9000 | miner debugging: faux-mining by theuni · Pull Request #9000 · bitcoin/bitcoin · GitHub
22:23 < cfields> but, as long as they're there, yea, i suppose so
22:24 < cfields> luke-jr: which reminds me, in the same vein, i think vbrequired needs some segwit love too?
22:24 < luke-jr> ?     
22:25 < cfields> luke-jr: i just glanced at it briefly tonight, vbrequired is hard-coded to 0 as far as i can tell?
22:25 < luke-jr> cfields: as it should be right now
22:25 < cfields> maybe i don't really understand what it's for, i just made a note to look deeper into it later
22:26 < cfields> (or ping you :p )
22:26 < gmaxwell> jonasschnelli: I think the GUI should say "X blocks behind" instead of "X days/weeks/months/years" -- the latter seems to be frequently misunderstood as how long the sync is going to take. .. or perhaps we could do something smarter than that.
22:26 < luke-jr> cfields: it's a constraint on the miner, vbs they *must* signal
22:27 < luke-jr> gmaxwell: the master/0.14 GUI currently has a real ETA
22:27 < cfields> luke-jr: ah, ok. I thought it was vbs they must understand. nm, then.
22:28 < luke-jr> cfields: that's "rules", since when they must understand it, there is no bit assigned anymore ;)
22:28 < cfields> luke-jr: yep, makes sense
22:29 < luke-jr> huh, serializing CTransaction can modify it? :o
22:29 < cfields> luke-jr: sorry, didn't mean to hijack. But we may as well fix up proposals if there's a use-case
22:29 < luke-jr> (it resizes the witness vector to match the input vector)
22:30 < luke-jr> cfields: well, I guess it's either that and/or make Eloipool submit witness-form gen tx
22:41 -!- DigiByteDev [~JT2@n218250011174.netvigator.com] has quit [Quit: DigiByteDev]     
22:57 < wumpus> gmaxwell: it used to say 'N blocks behind' a long time ago, was switched after lots of people complaining that blocks aren't a good indication of anything, though I'm not sure the time is that useful either...
22:58 < wumpus> gmaxwell: there's been so much iteration on that damn progress bar at some point :p
22:58 < gmaxwell> I was thinking that as I typed it...
22:58 < wumpus> oh I remember now, that complaint was more like 'blocks are too technical!'
22:59 < wumpus> 'show only blocks count in the debug interface and hover' well okay
22:59 < gmaxwell> luke-jr: I was just looking at gui in master... got 'n days behind'. which sounds like how long it will take.
22:59 < wumpus> maybe things changed with the blockchains hype
23:00 < luke-jr> gmaxwell: so the ETA doesn't distract enough from it, you're saying? :x
23:00 < gmaxwell> To be honest it might as well say "working hard at catching up! here is a spinner /-\_/-\..."
23:00 < luke-jr> lol     
23:00 < gmaxwell> luke-jr: I didn't see it! :P
23:00 < wumpus> here's dancing hamsters...
23:00 < luke-jr> I wasn't even looking for it, yet saw and appreciated it :p
23:01 < gmaxwell> but I've seen two reddit threads recently where they thought the x years behind was the catchup time, and one person on IRC. ... lemme go look 
23:01 < sipa> use this early in the chain when we don't have a good estimate yet: ¯\_(ツ)_/¯
23:01 < wumpus> yes I believe you that people get confused about it
23:01 < luke-jr> yeah, Core 0.13 doesn't have it yet
23:02 < luke-jr> I once had a feature request for Knots to have a button to click to play an audible "to the moon" sound bite..
23:02 < sipa> luke-jr: fixed in 8580
23:02 < wumpus> it's not really telling what it is doing either, e.g. if it was more apparent that it was validating history then in that context 'X years behind' makes sense
23:03 < luke-jr> someone want to make an animation of it checking the blocks? :p
23:03 < gmaxwell> okay the progress popup box I just dismissed. maybe now that we have that we don't need the progress bar to say 'foo behind'?
23:03 < wumpus> then again the new modal overlay with ETA and such is really nice and mitigates these concerns
23:03 < luke-jr> gmaxwell: I suggest change the progress bar to "Synchronising (ETA <x>)" ☺
23:03 < wumpus> luke-jr: +1
23:04 < fanquake> I agree, the new overlay is a good improvement.
23:04 < gmaxwell> luke-jr: that would be fine.
23:04 < wumpus> luke-jr: haha an animation of someone retrieving books of transactions from a chain of blocks and checking them?
23:05 -!- Alina-malina [~Alina-mal@37.157.216.188] has quit [Changing host]     
23:05 -!- Alina-malina [~Alina-mal@unaffiliated/alina-malina] has joined #bitcoin-core-dev     
23:05 < fanquake> wumpus or gmaxwell, have you seen any Boost related memory leaks during your testing?
23:05 < gmaxwell> the popup is kind of a sea of grey on my screen. 
23:05 < fanquake> Sorry to change topic, but I seem to have fallen down some memory leaks rabbit holes.
23:05 < gmaxwell> fanquake: no!
23:06 < wumpus> fanquake: nope.
23:06 < wumpus> fanquake: qt and ancillary GUI libs (Fontconfig etc) is the only thing leaking here
23:06 < wumpus> and all the leaks apparent in other libraries downstream seem to come from there
23:06 < fanquake> Hmm ok you see leaks from leveldb as well?
23:06 < wumpus> no     
23:07 < luke-jr> wumpus: I was thinking more of a horizontal line of pages sized based on the block size, and a red line scanning over them ;)
23:07 < wumpus> nothing in the core code
23:07 < gmaxwell> No. There is a little bit from the openssl RNG.
23:07 < wumpus> bitcoind is 100% leak clen
23:07 < gmaxwell> but bitcoind itself is 100% leak clean for me.
23:07 < luke-jr> wumpus: valgrind thinks the wallet code leaks
23:07 < wumpus> (at least the normal exit path, I haven't tried all kinds of panic shutdown scenarios ofc)
23:08 < luke-jr> it was around some BDB mess, so I didn't look far
23:08 < luke-jr> (one time at startup)
23:08 < wumpus> luke-jr: not seeing that here. But I use the system berkeleydb in Ubuntu 16.04
23:08 < wumpus> not 4.8 or that shit :)
23:09 < luke-jr> I use system 4.8 :P
23:09 < fanquake> Ok, I'll post a few details in a sec.
23:10 < wumpus> I honestly don't even know what version it is... lemme check. 5.3.
23:11 < wumpus> in any case I don't think we can ever prevent all one-time leaks in the GUI code, too many moving parts
23:12 < gmaxwell> they're mostly interesting to fix because sometimes they're actual bugs, and keeping down the inconsequential ones avoids hiding the actual bugs.
23:12 < wumpus> the only reason they're annoying at all is that they clutter the overview, so you may miss repeated leaks
23:13 < wumpus> right - I found an actual (minor) bug due to it with the rpc console thread
23:14 < fanquake> What I'm seeing seems to be Boost related, somewhere in CCoinsViewCache::FetchCoins()
23:15 < wumpus> do you see it with only the GUI, or with bitcoind too?
23:15 < wumpus> (not seeing them in any case but maybe an OSX thing?)
23:17 < fanquake> Have only tested with bitcoin-qt so far, I'll look at bitcoind now.
23:22 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-yxcnmjaozpbjrxnr] has joined #bitcoin-core-dev     
23:23 < fanquake> Here's a stack trace for the Boost leak, http://pastebin.com/V2QY8NT4 , not seeing anything on bitcoind so far (other than something related to berkely-db)
23:23 < wumpus> the most sneaky leak (and also most bytes wasted) I fixed in #9190 was the openssl certstore one (https://github.com/bitcoin/bitcoin/pull/9190/commits/4b2c2888868cb49806cd734ea36e7f5b032c3b00) . OpenSSL's documentation is somewhat fuzzy about it, but the assumption there was wrong, that X509_STORE_add_cert really tkaes ownership
23:23 < gribble> https://github.com/bitcoin/bitcoin/issues/9190 | qt: Plug many memory leaks by laanwj · Pull Request #9190 · bitcoin/bitcoin · GitHub
23:25 < wumpus> fanquake: what version of boost?
23:25 < sipa> fanquake: is that with a clean shutdown?
23:25 < fanquake> Boost 1.62.0
23:26 < fanquake> That is without shutting down, just while it's running.
23:26 < jonasschnelli> fanquake: I'm also compiling against Qt 5.7
23:26 < wumpus> you can only see *leaks* after you've shut down
23:26 < jonasschnelli> though I don't see the CCoinsViewCache::FetchCoins() leak
23:27 < wumpus> profilers can't predict the future of what an application is going to do with some memory, so you can't be sure some memory won't be released until the end of the shutdown sequence
23:28 < jonasschnelli> wumpus: I guess some leaks are detectable during runtime (when all pointers have lost the object)... not?
23:28 < wumpus> (although it could use some heuristic like "does the application retain any pointers to it" but that's horribly imprecise at most for C++)
23:28 < wumpus> jonasschnelli: not in general for unmanaged languages
23:29 < wumpus> and managed languages tend to have garbage collectors anyway :-)
23:29 < jonasschnelli> I guess it often leads to wrong detected leaks... (maybe losts).
23:29 < luke-jr> wumpus: I think it can be done with sufficient accuracy that the alternative is undefined behaviour?
23:29 < luke-jr> or at least very bad code :p
23:30 < jonasschnelli> I also think the Leak detector fanquake and I are using (Apples "Instruments") are not really made for C++... It's perfect when analyzing objective-C code.
23:30  * luke-jr actually finds runtime leak detection more useful than after exit, since many things use data but don't clean it up as well
23:30 < fanquake> Ok, I'm probably using the wrong terminology here then. I don't have too much experience using these kind of tools.
23:30 < wumpus> luke-jr: well that depends on your definition of 'bad code' I suppose, I'm just talking about theoretic properties not judging any particular use of cpu cycles :p
23:30 < fanquake> jonas Yes, hat might also be a part of the problem.
23:31 < luke-jr> wumpus: by bad code, I mean serializing the pointer in a uint8_t array or something
23:31 < luke-jr> or XORing it
23:31 < bitcoin-git> [bitcoin] jonasschnelli pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/0c577f2638b7...e4dbeb94998d
23:31 < bitcoin-git> bitcoin/master 76af4eb Jonas Schnelli: [Qt] fix coincontrol sort issue
23:31 < bitcoin-git> bitcoin/master 4231032 Wladimir J. van der Laan: [Qt] Clean up and fix coincontrol tree widget handling...
23:31 < bitcoin-git> bitcoin/master e4dbeb9 Jonas Schnelli: Merge #9185: [Qt] fix coincontrol sort issue...
23:31 < jonasschnelli> fanquake: But I was also surprised how many leaks I found with "Instruments" when writing pure C code.
23:31 < jonasschnelli> Though, you need a sleep(100) at the end of your application/unit-tests.
23:32 < jonasschnelli> Brews Valgrind is pretty broken on OSX:
23:32 < jonasschnelli> s/:/.
23:32 < luke-jr> LLVM also has a leak sanitizer
23:32 < fanquake> jonass As far as I know it won't work at all for a while https://bugs.kde.org/show_bug.cgi?id=365327
23:32 < fanquake> Neither does GDB
23:32 < wumpus> luke-jr: yes but I don't think e.g. pointer bit flipping tricks are by definition bad code, it just depends on the context, the resources available etc
23:33 < jonasschnelli> IMO all the CDB (BDB) leaks reported by "Instruments" are wrong reports... 
23:33 < fanquake> jonasschnelli so when we're looking at anything CWallet related in instruments (which is all I see looking at bitcoind) what are we seeing exactly?
23:34 < fanquake> heh, just what I was wondering.
23:34 < jonasschnelli> fanquake: tons of BDB leaks... 
23:34 < jonasschnelli> I guess >10MB
23:34 < jonasschnelli> (for a 300tx regtext wallet)
23:34 < wumpus> PSA: the macosx tools are wasting your time
23:34 < wumpus> :-)     
23:34 < fanquake> :o     
23:34 < jonasschnelli> wumpus: there is also the time profiler in "Instrumenst"... this is really cool.
23:34 < jonasschnelli> gpref like
23:35 < jonasschnelli> gperf
23:35 < jonasschnelli> wumpus: how did you detected the Qt leaks? IMO last time I used it on Bitcoin-Qt on Ubuntu I had to force shutdown my machine.. :)
23:36 < wumpus> luke-jr: yes - LLVM sanitizers that's what I used in #9190, built from latest LLVM git even, I have to track that anyhow for some GPU stuff
23:36 < gribble> https://github.com/bitcoin/bitcoin/issues/9190 | qt: Plug many memory leaks by laanwj · Pull Request #9190 · bitcoin/bitcoin · GitHub
23:36 < wumpus> jonasschnelli: ^
23:36 < jonasschnelli> Yes. Thanks...
23:37 < wumpus> CPPFLAGS="-ggdb -fsanitize=address -fno-omit-frame-pointer"   LDFLAGS="-fsanitize=address"
23:38 < fanquake> So we can ignore all of the certificate related Qt stuff as well?
23:39 < wumpus> it generates really nice rainbow console output, makes finding a use-after-free bug almost enjoyable
23:39 < wumpus> fanquake: not sure about ignoring anything, but please if you report leaks make sure that you are talking about the allocation residue after quitting the application
23:40 < fanquake> wumpus: Yep, have a bit better handle on this now. Looks like using some tools other than the native OS X stuff could be the way to go.
23:43 < wumpus> jonasschnelli: but yes visual profilers can be really cool, at a previous employer for solaris we had one that showed exactly what an application was doing on each core in a timeline overview, that was really useful for checking efficiency of concurrency etc
23:44 < wumpus> I kind of miss that in the linux toolset but I may be missing something, so many different tools and instrumentations
23:47 -!- baldur [~baldur@pool-100-2-139-91.nycmny.fios.verizon.net] has quit [Ping timeout: 258 seconds]     
23:48 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-blwocbacvfweoxah] has joined #bitcoin-core-dev     
23:51 -!- wasi [~wasi@gateway/tor-sasl/wasi] has joined #bitcoin-core-dev     
23:52 -!- wasi [~wasi@gateway/tor-sasl/wasi] has quit [Remote host closed the connection]     
23:52 -!- wasi [~wasi@gateway/tor-sasl/wasi] has joined #bitcoin-core-dev     
23:55 < sipa> dtrace?
23:58 < wumpus> yes I think it was based on that