--- Log opened Wed Apr 03 00:00:34 2019
00:03 < gwillen> wumpus: ... actually it appears to be pretty broken with multiple wallets
00:03 < gwillen> I never thought to try that
00:04 < gwillen> it seems to keep UTXOs selected when switching wallets that don't exist in the second wallet
00:04 < gwillen> I'm not sure what happens if I then try to spend them.
00:04 < gwillen> perhaps this is the secret backdoor way to spend from multiple wallets at once, heh
00:05 < gwillen> anyway I am tempted to fix it but I don't have spare cycles right now, ask me again in a month or two and I'll probably do it
00:07 -!- Guyver2 [AdiIRC@guyver2.xs4all.nl] has joined #bitcoin-core-dev
00:14 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has joined #bitcoin-core-dev
00:14 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
00:16 -!- fanquake [~fanquake@unaffiliated/fanquake] has quit []
00:27 < wumpus> gwillen: please create an issue for it at least
00:29 < wumpus> fanquake: thanks for keeping an eye on changes to the release notes-if this keeps up we'll have to restrict edits to people in the bitcoin-core org (I'd prefer not to as this loses some potential documentation-only contributors), but this may well be accidental and they fixed it
00:49 -!- fanquake [~fanquake@unaffiliated/fanquake] has joined #bitcoin-core-dev
00:57 < gwillen> wumpus: hmm, this may be either caused or revealed by my changes, investigating before I file it
01:00 < jonasschnelli> cfields: 0.18.0rc3 osx detached signatures are here: https://github.com/bitcoin-core/bitcoin-detached-sigs/pull/24
01:00 < gmaxwell> I've been watching logs on a couple nodes and it looks like new installs of 0.16.1 are significantly more common than later versions.
01:01 < gmaxwell> I wonder if there is some outdated link someplace.
01:01 < wumpus> maybe some distro did the cursed thing
01:01 < gmaxwell> (or there is some spy/sybil-attacker that looks like these things, but I don't think so as I'm already pretty heavily excluding things that look like that)
01:01 < wumpus> (e.g. freeze one version of bitcoin core as "stable")
01:03 < gmaxwell> Checking my logs most of them appear to be IPs in china.
01:03 < gmaxwell> jl2012: ^ any thoughts for this? is it possible some high visibility chinese language bitcoin resource links to 0.16.1 ?
01:07 < gmaxwell> looks like 98% of them are chinese.
01:09 < jl2012> I'll take a look
01:10 < jl2012> Are they from random ip, or from a few datacenters?
01:10 -!- setpill [~setpill@unaffiliated/setpill] has joined #bitcoin-core-dev
01:12 < gmaxwell> Many, though a large number are from 1.196.144.0/24
01:13 -!- setpill [~setpill@unaffiliated/setpill] has quit [Client Quit]
01:18 -!- setpill [~setpill@unaffiliated/setpill] has joined #bitcoin-core-dev
01:24 < gwillen> wumpus: filed #15725, I can't make this happen every time but it's reliable enough that I'm convinced it's probably not anything I touched
01:24 < gribble> https://github.com/bitcoin/bitcoin/issues/15725 | Manual coin control dialog interacts badly with multiple wallets · Issue #15725 · bitcoin/bitcoin · GitHub
01:26 < wumpus> gwillen: thank you
01:36 -!- Zenton [~user@unaffiliated/vicenteh] has joined #bitcoin-core-dev
01:51 -!- timothy [~tredaelli@redhat/timothy] has joined #bitcoin-core-dev
02:07 < fanquake> Added a validating subtree merges how-to to core-review: https://github.com/fanquake/core-review/blob/master/subtree-merge.md
02:09 < fanquake> Also, anyone is upgrading to Xcode 10.2, you may have to re-install the macOS_SDK_headers.pkg, otherwise your depends builds may break.
02:25 -!- kryoha [1838fc28@gateway/web/freenode/ip.24.56.252.40] has joined #bitcoin-core-dev
02:38 < wumpus> fanquake: this is very useful, thanks for working on that document
02:41 < wumpus> I think we also need to document the exact steps to update the various subtrees, there's some overlap with the "Subtrees" section in doc/developer-notes.md btw
02:47 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
02:47 < bitcoin-git> [bitcoin] ajtowns opened pull request #15726: bitcoin-cli -yaml support (master...201904-cli-yaml) https://github.com/bitcoin/bitcoin/pull/15726
02:47 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
02:57 -!- kryoha [1838fc28@gateway/web/freenode/ip.24.56.252.40] has quit [Ping timeout: 256 seconds]
03:01 < wumpus> yaml?!?
03:04 -!- Nakamoto [5089a2e0@gateway/web/freenode/ip.80.137.162.224] has joined #bitcoin-core-dev
03:12 -!- nullptr| [~nullptr|@ip-94-112-134-45.net.upcbroadband.cz] has quit [Ping timeout: 246 seconds]
03:13 < fanquake> wumpus Yea just documenting useful stuff. All the things I do infrequently enough that I have to go look it/the process up again.
03:18 -!- nullptr| [~nullptr|@ip-94-112-134-45.net.upcbroadband.cz] has joined #bitcoin-core-dev
03:18 < aj> wumpus: i can't read numbers without thousands separators man
03:19 -!- jonatack [58aba822@gateway/web/freenode/ip.88.171.168.34] has joined #bitcoin-core-dev
03:19 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...]
03:29 -!- Nakamoto [5089a2e0@gateway/web/freenode/ip.80.137.162.224] has quit [Quit: Page closed]
03:35 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
03:37 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev
04:16 -!- michaelfolkson [~textual@146.185.56.214] has joined #bitcoin-core-dev
04:43 -!- zivl [~zivl@unaffiliated/zivl] has quit [Quit: zivl]
04:48 -!- jonatack [58aba822@gateway/web/freenode/ip.88.171.168.34] has quit []
04:56 -!- zivl [~zivl@unaffiliated/zivl] has joined #bitcoin-core-dev
04:57 -!- michaelfolkson [~textual@146.185.56.214] has quit [Quit: Sleep mode]
05:00 -!- emzy [~quassel@unaffiliated/emzy] has quit [Ping timeout: 258 seconds]
05:03 -!- emzy [~quassel@raspberry.emzy.de] has joined #bitcoin-core-dev
05:04 -!- emzy [~quassel@raspberry.emzy.de] has quit [Client Quit]
05:08 -!- michaelfolkson [~textual@146.185.56.214] has joined #bitcoin-core-dev
05:09 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has joined #bitcoin-core-dev
05:10 -!- michaelfolkson [~textual@146.185.56.214] has quit [Client Quit]
05:13 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has quit [Ping timeout: 256 seconds]
05:17 -!- michaelfolkson [~textual@146.185.56.214] has joined #bitcoin-core-dev
05:27 -!- emzy [~quassel@raspberry.emzy.de] has joined #bitcoin-core-dev
05:31 -!- emzy [~quassel@raspberry.emzy.de] has quit [Changing host]
05:31 -!- emzy [~quassel@unaffiliated/emzy] has joined #bitcoin-core-dev
05:31 < luke-jr> FWIW, it looks like Boost::Process 1.45 is broken in that it doesn't recongised died-by-signal as exited (and throws exceptions wildly as a result)
05:31 < luke-jr> 1.47 has it fixed I guess
05:36 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has quit [Ping timeout: 256 seconds]
05:37 < fanquake> luke-jr Boost.Process in Boost 1.45 ? I thought it was only introduced in 1.64? Or are you talking about something else?
05:51 < luke-jr> sorry, I have my versions wrong, sec
05:51 < luke-jr> 1.65 vs 1.67
06:01 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has quit [Quit: Leaving]
06:12 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has joined #bitcoin-core-dev
06:13 -!- Karyon [~Karyon@unaffiliated/karyon] has quit [Ping timeout: 244 seconds]
06:14 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has joined #bitcoin-core-dev
06:16 -!- shtirlic [~shtirlic@ec2-18-185-133-40.eu-central-1.compute.amazonaws.com] has quit [Quit: ZNC - http://znc.in]
06:27 -!- michaelfolkson [~textual@146.185.56.214] has quit [Quit: Sleep mode]
06:28 -!- promag [~promag@bl16-114-47.dsl.telepac.pt] has joined #bitcoin-core-dev
06:33 -!- shtirlic [~shtirlic@ec2-18-185-133-40.eu-central-1.compute.amazonaws.com] has joined #bitcoin-core-dev
06:37 -!- Karyon [~Karyon@unaffiliated/karyon] has joined #bitcoin-core-dev
06:46 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
06:46 < bitcoin-git> [bitcoin] jnewbery opened pull request #15728: [wallet] Refactor relay transactions (master...2019_03_refactor_relay_transactions) https://github.com/bitcoin/bitcoin/pull/15728
06:46 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
06:47 -!- Aaronvan_ [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
06:50 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 250 seconds]
07:09 -!- michaelfolkson [~textual@146.185.56.214] has joined #bitcoin-core-dev
07:13 -!- michaelfolkson [~textual@146.185.56.214] has quit [Ping timeout: 268 seconds]
07:17 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...]
07:28 -!- Aaronvan_ [~AaronvanW@unaffiliated/aaronvanw] has quit []
07:35 -!- pinheadmz [~matthewzi@c-76-102-227-220.hsd1.ca.comcast.net] has quit [Quit: pinheadmz]
08:00 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev
08:02 < jl2012> gmaxwell: "Miner block size removed" in the release note of 0.16.1 led to some misunderstanding in China
08:02 < jl2012> some thought 0.16.1 was a block size hardfork
08:05 < jl2012> but those were results from June 2018. No recent discussions
08:06 < jl2012> and some people clarified that in Chinese at that time
08:31 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
08:31 < bitcoin-git> [bitcoin] promag opened pull request #15729: rpc: Ignore minconf parameter in getbalance (master...2019-04-drop-getbalance-minconf) https://github.com/bitcoin/bitcoin/pull/15729
08:31 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
08:43 -!- cheetah2 [~cheetah2@172.97.32.184] has joined #bitcoin-core-dev
08:45 < cheetah2> im getting error code -28 prunning blockstore with bitcoin-cli getinfo does it mean its crashed or is it working?
08:50 -!- pinheadmz [~matthewzi@208.69.41.109] has joined #bitcoin-core-dev
09:03 -!- cheetah2 [~cheetah2@172.97.32.184] has quit [Remote host closed the connection]
09:05 -!- setpill [~setpill@unaffiliated/setpill] has quit [Quit: o/]
09:09 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
09:10 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
09:10 < bitcoin-git> [bitcoin] promag opened pull request #15730: rpc: Show scanning details in getwalletinfo (master...2019-04-getwalletinfo-scanning) https://github.com/bitcoin/bitcoin/pull/15730
09:10 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
09:12 -!- Aaronvan_ [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
09:14 -!- Aaronvan_ is now known as AaronvanW_
09:15 -!- Tralfaz [49dde1e1@gateway/web/freenode/ip.73.221.225.225] has quit [Ping timeout: 256 seconds]
09:15 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 250 seconds]
09:20 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
09:20 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/2c364fde423e...ba54342c9dd3
09:20 < bitcoin-git> bitcoin/master fa292ad MarcoFalke: doc: rpc-mining: Clarify error messages
09:20 < bitcoin-git> bitcoin/master ba54342 MarcoFalke: Merge #15685: doc: rpc-mining: Clarify error messages
09:20 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
09:20 -!- joshmg [~textual@24.192.60.218] has joined #bitcoin-core-dev
09:21 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
09:21 < bitcoin-git> [bitcoin] MarcoFalke merged pull request #15685: doc: rpc-mining: Clarify error messages (master...1903-docMining) https://github.com/bitcoin/bitcoin/pull/15685
09:21 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
09:25 < MarcoFalke> proposedmeetingtopic: TODOs in 0.18.0 Release Notes Draft
09:31 < MarcoFalke> [05:41] <wumpus> I think we also need to document the exact steps to update the various subtrees, there's some overlap with the "Subtrees" section in doc/developer-notes.md btw
09:31 < MarcoFalke> fanquake: 
09:31 < MarcoFalke> Those exist in https://github.com/bitcoin/bitcoin/tree/ba54342c9dd3f2e5cdeed9ac57f1924f0d885cc6/test/lint#git-subtree-checksh
09:31 < MarcoFalke> and https://github.com/bitcoin-core/bitcoin-maintainer-tools/tree/ae4ed2a58d67764dbfe2dc1ba00667659e330090#subtree-updates
09:40 -!- pinheadmz [~matthewzi@208.69.41.109] has quit [Ping timeout: 255 seconds]
09:45 -!- pinheadmz [~matthewzi@208.69.41.109] has joined #bitcoin-core-dev
09:51 -!- AaronvanW_ [~AaronvanW@unaffiliated/aaronvanw] has quit []
09:59 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
09:59 -!- pinheadmz [~matthewzi@208.69.41.109] has quit [Ping timeout: 245 seconds]
10:03 < instagibbs> the bitcoin-maintainer one is the more useful one IIRC
10:05 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Remote host closed the connection]
10:06 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
10:17 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Remote host closed the connection]
10:18 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
10:21 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Remote host closed the connection]
10:22 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
10:22 < MarcoFalke> They serve different purposes. The "linter" is for checking and the maintainer-tool is for creating/bumping
10:24 < cfields> jonasschnelli: not sure if you've discussed already, but it looks like you pushed your rc2 gitian sigs as rc3 accidentally.
10:24 < cfields> Though your signature attached and verified correctly. So I assume you just pushed the wrong thing on accident.
10:25 < cfields> err
10:29 -!- shesek [~shesek@unaffiliated/shesek] has quit [Ping timeout: 250 seconds]
10:36 < cfields> jonasschnelli: yes, ok, ^^.
10:36 < cfields> I'm not going to worry about waiting for updated sigs, though. We have enough, and it looks like the correct thing was signed in the end.
10:39 < cfields> gitian builders: detached sigs are pushed for v0.18.0rc3.
10:42 -!- Zenton [~user@unaffiliated/vicenteh] has quit [Ping timeout: 245 seconds]
10:46 -!- morcos [~morcos@gateway/tor-sasl/morcos] has quit [Remote host closed the connection]
10:46 -!- morcos [~morcos@gateway/tor-sasl/morcos] has joined #bitcoin-core-dev
10:53 -!- pinheadmz [~matthewzi@208.69.41.109] has joined #bitcoin-core-dev
10:54 -!- ExtraCrispy [~ExtraCris@gateway/tor-sasl/extracrispy] has quit [Remote host closed the connection]
10:54 -!- ExtraCrispy [~ExtraCris@gateway/tor-sasl/extracrispy] has joined #bitcoin-core-dev
10:59 -!- pinheadmz [~matthewzi@208.69.41.109] has quit [Ping timeout: 255 seconds]
11:00 -!- pinheadmz [~matthewzi@208.69.41.109] has joined #bitcoin-core-dev
11:07 < jonasschnelli> cfields: oh. let me check
11:07 -!- pinheadmz [~matthewzi@208.69.41.109] has quit [Ping timeout: 250 seconds]
11:16 -!- pinheadmz [~matthewzi@208.69.41.109] has joined #bitcoin-core-dev
11:16 -!- pinheadmz [~matthewzi@208.69.41.109] has quit [Client Quit]
11:20 < jonasschnelli> cfields: yes. I uploaded the rc2 sigs for osx/win (not for linux though).
11:20 < jonasschnelli> Will fix
11:25 -!- dviola [~diego@unaffiliated/dviola] has joined #bitcoin-core-dev
11:39 -!- hebasto [~hebasto@95.164.65.194] has quit [Remote host closed the connection]
11:41 -!- d_t [~d_t@lexington23associates09.l.subnet.rcn.com] has joined #bitcoin-core-dev
11:45 -!- kexkey [~kexkey@68.168.115.53] has joined #bitcoin-core-dev
12:02 -!- shesek [~shesek@185.3.144.77] has joined #bitcoin-core-dev
12:02 -!- shesek [~shesek@185.3.144.77] has quit [Changing host]
12:02 -!- shesek [~shesek@unaffiliated/shesek] has joined #bitcoin-core-dev
12:03 -!- pinheadmz [~matthewzi@c-76-102-227-220.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
12:06 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Remote host closed the connection]
12:12 -!- jtimon [~quassel@59.31.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev
12:14 -!- owowo [~ovovo@unaffiliated/ovovo] has quit [Ping timeout: 245 seconds]
12:18 -!- beau [aafdbacb@gateway/web/freenode/ip.170.253.186.203] has joined #bitcoin-core-dev
12:19 -!- owowo [~ovovo@unaffiliated/ovovo] has joined #bitcoin-core-dev
12:19 -!- beau [aafdbacb@gateway/web/freenode/ip.170.253.186.203] has quit [Client Quit]
12:30 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
12:35 < wumpus> MarcoFalke: good to know, I think I knew about that one once but forgot about it
12:45 -!- Zenton [~user@unaffiliated/vicenteh] has joined #bitcoin-core-dev
12:52 -!- mn9495881 [~nodebot@cpe-67-243-195-224.nyc.res.rr.com] has joined #bitcoin-core-dev
13:04 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has joined #bitcoin-core-dev
13:08 -!- timothy [~tredaelli@redhat/timothy] has quit [Read error: Connection reset by peer]
13:49 -!- Guyver2 [AdiIRC@guyver2.xs4all.nl] has quit [Quit: Going offline, see ya! (www.adiirc.com)]
14:17 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has quit [Ping timeout: 256 seconds]
14:40 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
14:40 < bitcoin-git> [bitcoin] jamesob opened pull request #15735: Add scheduler deadlock detection (master...2019-04-serialize-scheduler) https://github.com/bitcoin/bitcoin/pull/15735
14:40 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
14:43 < gmaxwell> jl2012: there isn't a chinese version of bitcoin.org or something that only shows old links?  I'm just at a loss as to what would be causing people to start new nodes using old code.
14:45 -!- pinheadmz [~matthewzi@c-76-102-227-220.hsd1.ca.comcast.net] has quit [Read error: Connection reset by peer]
14:48 -!- pinheadmz [~matthewzi@c-76-102-227-220.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
14:49 < phantomcircuit> gmaxwell, are they all on aws or something?
14:50 < midnightmagic> gmaxwell: the TMSR people and their propaganda maybe?
14:51 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
14:52 < gwillen> sipa or someone: are descriptors supposed to accept "h" anywhere they would accept "'"?
14:53 < gwillen> gmaxwell: ^
14:54 < gwillen> it seems like maybe there is an untested interaction with checksums
14:54 < gwillen> I am finding that my descriptors with 'h' are being parsed as invalid
14:55 < gwillen> if I getdescriptorinfo on a descriptor that uses 'h', I get back one that uses "'" and has a valid checksum, but if I then replace the ' with h and keep the checksum the same, the resulting descriptor is rejected
14:56 < sipa> gwillen: yes, you can use either h or '
14:56 < sipa> and the checksum will depend on which one you use
14:56 < gwillen> hmmmmmm
14:56 < gwillen> but getdescriptorinfo canonicalizes from "h" to "'" _before_ computing the checksum
14:56 < gwillen> so I can't get one with h :-)
14:56 < sipa> yes
14:56 < gwillen> also it seems weird that they are treated differently, I would argue against that but maybe it's too late
14:56 -!- shtirlic [~shtirlic@ec2-18-185-133-40.eu-central-1.compute.amazonaws.com] has quit [Quit: ZNC - http://znc.in]
14:57 < sipa> gwillen: the alternative would require a checksum algorithm that's not black box
14:57 < gwillen> it would just require that you always canonicalize before checksumming
14:57 < gwillen> since it seems like the very first thing you do on parsing is canonicalize anyway
14:58 < sipa> canonicalizing requiring understand the descriptors
14:58 < sipa> while checksums can hopefully be performed by things that treat descriptors as strings
14:58 < sipa> it was probably a mistake to permit both h and ' in the first place, but it's far too late for that
14:58 < gwillen> this is somewhat sad for dev use, because the primary use of 'h' is not having to quote the apostrophe
14:58 < gwillen> but there is no easy way to get a checksum on a descriptor that uses h
14:58 < sipa> ah i see
14:58 < gwillen> so in practice you always have to use the apostrophe
14:59 < gwillen> which defeats the purpose of h existting
14:59 -!- joshmg [~textual@24.192.60.218] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
14:59 < sipa> well you don't have to use getdescriptorinfo
15:00 < gwillen> is there an RPC that will checksum a descriptor for me without altering it first
15:00 < sipa> not an RPC, but there is python code for it
15:00 < gwillen> can I argue for getdescriptorinfo not to alter the descriptor provided
15:00 < gwillen> (is there any other issue like this aside from h/'?)
15:01 < sipa> if you include a private key in the descriptor, the canonical version will drop it
15:01 -!- shtirlic [~shtirlic@ec2-18-185-133-40.eu-central-1.compute.amazonaws.com] has joined #bitcoin-core-dev
15:01 < sipa> because that's just syntactic sugar for the public version + simultaneously also conveying the private key
15:01 < gwillen> except again for the checksum caring about the difference
15:02 < sipa> yes
15:03 < gwillen> I am heavily tempted to disable descriptor checksums locally while I'm doing development to make this more usable
15:03 < gwillen> that is not a good sign :-P
15:04 < sipa> it wouldn't be hard to add an RPC that just adds the checksum, but otherwise leaves the descriptor untouched
15:05 < sipa> would that help?
15:05 < gwillen> that would be helpful, but I also wonder if getdescriptorinfo really needs to canonicalize the descriptor
15:05 < gwillen> but either one would do, yeah
15:05 < gwillen> although I'm now building with descriptor checksums disabled anyway, since it was a one-line change
15:05 < sipa> agree, perhaps it doesn't need to in the first place
15:05 < gwillen> *nods*
15:06 < gwillen> if the canonical version were 'h' rather than "'", it wouldn't require parsing to checksum the canonical version, since "'" can't appear anywhere else as far as I know
15:06 < gwillen> (but this doesn't solve private keys, and also it's surely too late.)
15:09 < sipa> a side effect of having the checksum depend on the exact string representation also means that 'canonical' version doesn't mean much
15:09 < sipa> we could change the canonical encoding without breaking anything
15:09 < gwillen> certainly h is easier to work with than "'"
15:09 < gwillen> I don't know if it has any downsides I'm not thinking of
15:10 < sipa> the ' notation is more widespread
15:10 < gmaxwell> I don't think the rpc turning ' into h would be a problem. It does need to accept ' in inputs, of course.
15:11 < gwillen> yeah, it should absolutely still accept both
15:11 < gwillen> right now it turns h into '
15:11 < sipa> wouldn't be hard to make it not touch it as well
15:12 < gwillen> hm, I just did a ranged descriptor import with a range of 10,000
15:12 < gwillen> and it seems to have hung
15:12 < gwillen> it's been 120 seconds
15:12 < achow101> I think having the canonical one be 'h' would be better
15:12 < sipa> is it rescanning?
15:12 < gwillen> I used {"rescan": false}
15:12 < sipa> that's ungood
15:13 < gwillen> I will attach a debugger
15:13 < achow101> also having getdescriptorinfo give the checksum for descriptors with privkeys too
15:14 < achow101> gwillen: I think that is not unexpected
15:14 < gwillen> no?
15:15 < achow101> IIRC it writes each generated key and key origin data independently without batching (I know, bad idea), so it takes a while due to disk I/O
15:15 < gwillen> I mean, awhile is awhile, but it's now been like 5 minutes
15:15 < gwillen> that is a long while to perform 10,000 writes
15:15 < gwillen> and the gui thread is totally dead now
15:15 < gwillen> before the GUI was responding but the console wasn't answering RPCs
15:16 < achow101> hmm, that might be hung and not just taking awhile
15:17 < gwillen> we are inside CWallet::CanGetAddresses
15:18 < achow101> gwillen: oh, deadlock?
15:18 < gwillen> oh, maybe, yeah, we're trying to lock a mutex
15:18 < achow101> the gui thread probably went into cangetaddresses but can't acquire the lock so it
15:18 < sipa> gwillen: is this 0.18/master or something with changes by you?
15:18 < achow101> as the lock is held by the rpc thread for import
15:18 < gwillen> sipa: changes by me
15:19 < gwillen> not to any of this though I would expect?
15:20 < gwillen> it looks like we may indeed be deadlocked trying to take cs_wallet at the top of CanGetAddresses
15:20 < gmaxwell> What lock would be held before cs_wallet in the call path to CanGetAddresses?  can you get a backtrace?
15:21 < gmaxwell> (so we can figure out what the other lock involved in the inversion is?)
15:21 < sipa> gwillen: compile with -DDEBUG_LOCKORDER
15:21 < sipa> and try to reproduce
15:21 < gmaxwell> I wonder if essentially anyone has used the GUI with -DDEBUG_LOCKORDER ?
15:21 < gwillen> let me paste the backtrace first for gmaxwell 
15:21 < gwillen> I will try with debug_lockorder next
15:21 < sipa> oh, the stack trace (for all threads) should actually tell you just as much
15:21 < gwillen> there are a lot of threads, how do I backtrace threads in bulk in lldb
15:21 < gmaxwell> thread apply all bt 
15:22 < gmaxwell> works in gdb, I assume lldb too
15:22 < gwillen> sadly it does not work in lldb
15:22 < gwillen> and I cannot gdb because mac
15:22 < sipa> sadness
15:22 < gmaxwell> thread backtrace all
15:22 < gmaxwell> bt all
15:22 < gwillen> aha
15:22 < gmaxwell> ^ lldb commands.
15:24 -!- shtirlic [~shtirlic@ec2-18-185-133-40.eu-central-1.compute.amazonaws.com] has quit [Quit: ZNC - http://znc.in]
15:24 < achow101> gwillen: can you check if your wallet file is being modified?
15:25 < gmaxwell> (being modified while not paused in the debugger)
15:25 < gmaxwell> GUI hanging in CanGetAddresses doesn't itself imply that there is a deadlock there. e.g. it could just be waiting on a lock that is deadlocked elsewhere, or not even locked but absurdly slow.
15:26 < gwillen> hm yes, it is being modified
15:26 < gwillen> it is growing.
15:26 < achow101> gmaxwell: I'm pretty sure it's hanging in CanGetAddresses because cs_wallet is locked by the inport, which CanGetAddresses needs to acquire
15:26 < gwillen> it's quite large.
15:26 < achow101> no shit. that's 10000 keys
15:26 < gwillen> how big is a key?
15:27 < gwillen> this seems like support for "not even locked but absurdly slow"
15:27 < gmaxwell> I was about to say that.
15:27 < achow101> 33 bytes + keyid a few times + metadata. probably like 80 bytes at least
15:27 < gmaxwell> key records are on the order of 100 bytes I think.
15:28 < achow101> i'm testing this locally. it looks like it spends a while on key derivation also
15:28 < gwillen> does it also duplicate full key origin info for each?
15:28 < gwillen> that would make it bigger.
15:28 < achow101> yes
15:28 < gwillen> we're at 4 megs and growing
15:28 < achow101> not unexpected
15:28 < gwillen> so it's definitely more than 80 bytes per.
15:28 < sipa> key entry + metqdata entry + keypool entry if applicable
15:28 < sipa> for each
15:29 < gwillen> I considered picking a more realistic number than 10,000, fwiw, I have no actual need for this to work
15:29 < sipa> also, uncompresed wallet?
15:29 < gwillen> I typed it mostly on a lark because computers are fast
15:29 < sipa> in that case private keys are 270 bytes
15:29 < gwillen> hm, I don't know, whatever the default is for a new wallet
15:29 < gmaxwell> well it should be fast this is a bug.
15:29 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Remote host closed the connection]
15:30 < gmaxwell> I mean if you said it took a couple seconds, sure whatever, who cares.
15:30 < gwillen> yeah
15:30 < achow101> I just tested it. it took 6 minutes for my machine to do an import of a ranged descriptor for 10000 keys
15:30 < gwillen> I should clearly go into QA
15:31 < gwillen> given my ability to provoke software to do stupid shit ;-)
15:33 < gwillen> filed #15739
15:33 < gribble> https://github.com/bitcoin/bitcoin/issues/15739 | large ranged descriptor imports are sloooooooooow. · Issue #15739 · bitcoin/bitcoin · GitHub
15:33 < gmaxwell> I'm gussing it's doing something stupid like opening and closing the wallet for each key.
15:33 < achow101> gmaxwell: it's not using a batch write. using a batch write should speed it up significantly
15:34 < sipa> gwillen: it absolutely is doing that
15:34 < gwillen> even opening and closing the wallet 10,000 times shouldn't take 6 minutes, should it?
15:34 < sipa> are you on an I/O starved VPS? :p
15:34 < gmaxwell> gwillen: it writes and fsyncs each time.
15:34 < gwillen> I guess if every time it opens it, it scans and validates a bunch of things, fires a bunch of notifications, etc. etc.
15:34 < gmaxwell> and on mac there is no file-centric fsync.
15:34 < achow101> working on fix
15:34 -!- joshmg [~textual@d149-67-39-69.try.wideopenwest.com] has joined #bitcoin-core-dev
15:35 -!- joshmg [~textual@d149-67-39-69.try.wideopenwest.com] has quit [Client Quit]
15:35 < sipa> achow101: nice
15:36 < sipa> i guess some of the wallet encryption logic can be used to avoid opening/closing the db every time?
15:36 < gwillen> will my testnet wallet get rekt if I murder this process in the middle of whatever it's doing
15:37 < gmaxwell> We had this same problem with the original wallet creation at one point, IIRC
15:37 < gwillen> sipa: what's the deal with wallet compression, is it an option?
15:38 -!- shtirlic [~shtirlic@ec2-35-156-179-128.eu-central-1.compute.amazonaws.com] has joined #bitcoin-core-dev
15:39 < sipa> gwillen: if you never encrypted the wallet it's unencrypted
15:40 < gwillen> erm, you said compression before, but this time you said encryption
15:40 < sipa> i meant encrypted :)
15:40 < gwillen> oh, heh
15:40 < gwillen> does encryption also compress
15:40  * sipa falls asleep
15:41 < gwillen> or is this not actually relevant to the problem of walletfile being huge
15:41 < gwillen> you said something about 270 bytes per key
15:41 < gwillen> I would have assumed default compression, especially if we're repeating mostly-redundant key origin info for every key
15:41 < gwillen> that should be extremely compressible
15:42 < gwillen> my wallet has reached 8.5 megs 
15:42 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...]
15:42 < sipa> gwillen: the old unencrypted wallet format stores private keys in a ricldiculous der encoded format of 279 bytes per key
15:43 < sipa> where only 32 bytes are needed
15:43 < gwillen> ah, well this wallet should be relatively new, like within the last few months
15:43 < sipa> not relevant
15:43 < sipa> the encoding used for unencrypted keys is still that der one
15:43 < gwillen> also this is watchonly so there are no private keys anyway
15:43 < sipa> oh!
15:43 < sipa> then this is all irrelevant
15:44 < gwillen> so presumably all that space is being taken up by key origin info and other metadata
15:44 < gwillen> although that's still more than 1kb per key, which seems huge
15:44 < gwillen> achow101: how large did your wallet get after you spent 6 minutes importing 10,000 keys
15:45 < sipa> gwillen: doesn't surprise me
15:45 < achow101> gwillen: 8 MB, but I also did it again and now it's 16 MB
15:45 -!- Randolf [~randolf@24.114.43.105] has joined #bitcoin-core-dev
15:45 < gwillen> heh
15:45 < gwillen> so mine was almost done probably
15:47 < gwillen> doing 100 takes 3 seconds on my machine, so 10000 would have taken about 5 minutes to finish
15:55 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev
16:06 < achow101> well batching writes reduced that from 6:56 to 1:24
16:08 < gwillen> ha
16:18 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
16:18 < bitcoin-git> [bitcoin] achow101 opened pull request #15741: Batch write imported stuff in importmulti (master...batch-write-importmulti) https://github.com/bitcoin/bitcoin/pull/15741
16:18 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
16:18 < achow101> gwillen: ^^
16:19 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit []
16:21 < gwillen> :+1: will review!
16:27 < gwillen> achow101: is this kind of manual batching typical of using WalletBatch? I wonder if it would make sense to have a self-flushing WalletWriteCache or something 
16:27 < gwillen> to avoid having to have all the fiddly counting of entries
16:28 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has quit [Ping timeout: 250 seconds]
16:28 < achow101> gwillen: we do something similar for initial key generation
16:28 < achow101> i was mostly following upgradekeymetadata (aka copy paste) which does counting
16:30 < gwillen> would be good to factor it out nicely ;-)
16:30 -!- Randolf [~randolf@24.114.43.105] has quit [Quit: Leaving]
16:32 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has joined #bitcoin-core-dev
16:34 -!- dviola [~diego@unaffiliated/dviola] has quit [Quit: WeeChat 2.4]
16:39 -!- d_t [~d_t@lexington23associates09.l.subnet.rcn.com] has quit [Ping timeout: 250 seconds]
16:46 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
16:46 < bitcoin-git> [bitcoin] kallewoof closed pull request #13746: -masterdatadir for datadir bootstrapping (master...masterdatadir) https://github.com/bitcoin/bitcoin/pull/13746
16:46 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
16:47 -!- dqx_ [~dqx@unaffiliated/dqx] has quit [Remote host closed the connection]
16:50 -!- fanquake [~fanquake@unaffiliated/fanquake] has quit []
16:58 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has joined #bitcoin-core-dev
16:59 -!- DeanWeen is now known as DeanGuss
17:02 < sipa> achow101: aftee batching the writes, it should be almost entirely cpu bound
17:04 -!- supay [dfe622a7@gateway/web/freenode/ip.223.230.34.167] has joined #bitcoin-core-dev
17:04 < sipa> so you're now at 8ms of CPU per generated key?
17:04 < supay> i lost access to my bitcoin and scrubbed my system for any data i could find. found some bip32/44 paths. trying to import private keys into bitcoin core using importprivkey, but that isn't working. there aren't any errors either. any help?
17:04 < sipa> oh, if we'd cache intermediary results in the key derivation we'd save a bunch
17:05 -!- drexl [~drexl@188.166.71.168] has joined #bitcoin-core-dev
17:07 < achow101> sipa: I'm pretty sure it's still spending most of the time on writing to the wallet as the wallet file starts growing in size almost immediately and that should only happen only after all keys are derivec
17:08 < gmaxwell> supay: what exactly does "isn't working" mean?
17:09 < supay> gmaxwell: i've done: bitcoin-cli importprivkey the_key "" true  -- and i wait about 25 mins or so for the rescan, and when i run bitcoin-cli getwalletinfo, the balances are still 0
17:10 < sipa> supay: are you sure the private key you're importing corresponds to the address?
17:10 < gwillen> hey supay, I assume you are 0x23212f from github?
17:10 < sipa> could it be you're importing a root key instead of the derived key?
17:11 < sipa> or gotten the compression flag wrong? (depending on how you obtained the private key)
17:11 < supay> sipa: to which address? i gather importing the private key suffices, as it has the address?
17:11 < supay> gwillen: yes, that is correct! :)
17:11 < sipa> supay: well i don't know where you got the key
17:11 < gwillen> supay: you said the path is m/44'/0'/0'/0/427 -- are you trying to import the master privkey, or the privkey derived from that path?
17:11 < supay> gwillen: thank you for your reply there. i wasn't aware this channel exists.
17:12 < gwillen> np, I didn't want to encourage you to come here since it's not really a tech support channel, but the people here are trustworthy :-)
17:12 < supay> gwillen: i have a bunch of lines like this one (multiple paths). i see them as 4 comma separated values. the 2nd column is address, the 4th is private key. i'm not sure if it's the master or otherwise
17:12 < gwillen> if you are trying to import the master privkey, that is your problem -- bitcoin core does not know about bip 44, and won't try to derive the child keys automatically
17:13 < gwillen> ahh, if each line has a different key, then I expect it is the child key, which I would expect to have the funds
17:13 < supay> that is correct, each line has a different key
17:13 < supay> wait, let me share some with 0 balance
17:13 < supay> is that safe?
17:14 < achow101> no, it is not safe
17:14 < gwillen> supay: if you "getaddressinfo" the address after importing the privkey, what does it say about ismine and solvable?
17:14 < gwillen> supay: sharing private keys is, to a first approximation, never ever safe
17:14 < supay> haha, makes sense
17:14 < supay> let me check getaddressinfo, one moment
17:15 < gwillen> and in fact, sharing private keys which are derived from a path related to other private keys that have funds is _specifically_ unsafe
17:15 < supay> gwillen: ismine for that address is current false
17:15 < gmaxwell> supay: rescan can take hours depending on your system, but the call blocks while it's in progress so you'll now when its done.
17:15 < gwillen> and the fact that this is not obvious results in some serious arguments over usability
17:15 < gwillen> gmaxwell: he said in his github issue that the rescan completed
17:15 < supay> gmaxwell: i've been observing debug.log. i'm sure rescan is complete
17:16 < gmaxwell> supay: good!
17:16 < supay> gmaxwell: wait, is ismine: false a good thing? :o
17:16 < gmaxwell> aside the above exchange with supay asking if it was safe to share zero balance keys with us in here, ... ahem,  non-hardened derrivation, ... how many users would even ask if it was safe? [/obrant]
17:17 < gwillen> gmaxwell: yes, indeed, the very fact that he asked the question is strong evidence of your rightness in this argument :-)
17:17 < supay> i'm so confused. i shouldn't have asked? 
17:17 < gwillen> haha, no, not at all
17:17 < supay> :D
17:18 < gwillen> just that most people would go ahead and share without asking
17:18 < gwillen> which would be bad because it would compromise their wallet
17:18 < achow101> supay: ismine false means that address is not part of your wallet
17:18 < supay> gwillen: ah yes, but you cautioned me in the github reply!
17:18 < gwillen> and there is an ongoing argument over the fact that this happens
17:18 < gmaxwell> supay:  it's good you asked, ignore my comment there-- I'm just pointing out that its relevant to a unrelated longer term debate.
17:18 < achow101> supay: so either importprivkey is not working (we have tests, but it's possible) or the address you have is not an address corresponding to that private key
17:18 < supay> achow101: that is correct. all the material i have read indicates directly using importprivkey. do i need to first add the address to the wallet?
17:19 < supay> gmaxwell: ah hehe
17:19 < gwillen> supay: if you run dumpprivkey address, it will show you what it thinks the privkey is for the address
17:19 < gwillen> and you can compare it to the one you have in your own dump
17:19 < supay> achow101: right, i really hope it is not the latter
17:19 < gmaxwell> I'm guessing that the privkey he is importing doesn't match the address he expects. Maybe due to a compressed flag?
17:19 < gwillen> if it's different, don't panic yet, there are lots of ways that the UX can be confusing
17:20 < gmaxwell> supay: what software wrote this list of addresses / paths/ keys?
17:20 < supay> gmaxwell: can't be very sure! :(
17:20 < achow101> supay: can you try `bitcoin-cli listaddressgroupings` and see if your address is in there?
17:20 < supay> i just searched for these patterns on a backup of my computer
17:20 < supay> achow101: sure, one moment
17:21 < gmaxwell> I believe supay said above he was getting ismine false on an address he believed he imported a private key for
17:21 < gmaxwell> If so, I think that proves that he did not successfully import a matching private key. 
17:21 < supay> achow101: hm, interesting. another one that i tried some time ago is on listaddressgroupings. this one, i ran importaddress but didn't import the private key yet
17:21 -!- fanquake [~fanquake@unaffiliated/fanquake] has joined #bitcoin-core-dev
17:21 < achow101> supay: that's expected. importaddress imports the address to be watched.
17:22 < supay> gmaxwell: is that the only conclusion? :(
17:22 < gwillen> supay: ahhhh sorry, did you run 'getaddressinfo' on one where you just imported the address? I meant to suggest you do it on one where you imported the key.
17:22 < supay> achow101: ah, alright
17:22 < achow101> do you see any unfamiliar addresses there? if so, try dumpprivkey on those addresses and see if you can find the one that matches the private key you imported
17:22 < supay> gwillen: ah no, i ran it on the one where i imported the key
17:22 < gwillen> supay: don't panic yet, there are many many ways that the user interface can go wrong, and most of them do not involve you losing coins once we figure out what's goin gon
17:22 < supay> achow101: i do infact, this was a clean install. there were definitely some new addresses
17:22 < supay> gwillen: oh! that's a relief
17:23 < gwillen> honestly all this stuff is very arcane
17:23 < gmaxwell> My leading hypothesis is that the dump he has shows the wrong addresses with the keys. E.g. because it mishandles the compressed flag.
17:23 < gmaxwell> Or because of some off by one, like each line is the address for the private key on the prior/next line.
17:24 < gmaxwell> either case is easily fixable if they're the issue.
17:24 < achow101> gmaxwell: i'm trying to figure out if he's got an off by one or something like that. once he can find the address that was imported for the private key, he can check if that's in his list
17:24 < gmaxwell> supay: has any of the addresses which have balances that you're trying to import spent before?
17:25 < achow101> gmaxwell: i would be surprised if it was a compressed flag problem. the key is compressed (he says it begins with K) and it's BIP 32 derived, which basically requires compressed keys
17:25 < supay> gmaxwell: not sure, but pm'd you one such address
17:25 < gmaxwell> achow101: darn, I missed the fact that we'd already asked for the leading character.
17:26 < supay> achow101: yep, starts with a K. another with a K. and there's one with an L
17:26 < achow101> gmaxwell: not so much asked but provided in the issue :) #15736
17:26 < gribble> https://github.com/bitcoin/bitcoin/issues/15736 | Unable to importprivkey · Issue #15736 · bitcoin/bitcoin · GitHub
17:28 < gmaxwell> supay: does the file you're taking these from have any kind of header that I could use to determine the originating software?
17:28 < gmaxwell> achow101: still possible some broken app was using bip32 with uncompressed keys. But I agree thats a lot less likely.
17:29 < supay> it had something indicating the columns. that's all i remember. unfortunately, got rid of the header!
17:29 < supay> that's how i know the last column was private key and second column was address
17:29 < gmaxwell> in the future when you're trying to recover data, take care to not throw anything out. :P
17:29 < supay> yess :(
17:30 < gmaxwell> do we really have no rpc that will take a private key and return addresses?
17:30 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
17:30 < achow101> gmaxwell: nope
17:31 < supay> gmaxwell: interestingly, i think electrum does that
17:31 < supay> when you import a private key, it says "so and so addresses were added"
17:32 < gmaxwell> probably the easiest thing to do would be to import a single additional key from the middle of the list with rescan false, and then look in dumpwallet output to see what new address got added.
17:33 < supay> oh, i see, let me try that
17:33 -!- IGHOR [~quassel@93.178.216.72] has quit [Read error: Connection reset by peer]
17:33 < gwillen> (in general I recommend always using rescan false, and then call rescanblockchain when you're ready to rescan -- you can give it a height to start scanning so it doesn't need to start in 2009)
17:35 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Ping timeout: 250 seconds]
17:36 < supay> oh, i didn't know you could do that! interesting :D
17:36 < supay> gmaxwell: my dumpwallet output is huge. so many keys, addresses etc. no idea where all of these came from
17:37 < achow101> supay: bitcoin core's default wallet will generate 2000 keys automatically
17:37 -!- IGHOR [~quassel@93.178.216.72] has joined #bitcoin-core-dev
17:37 < gmaxwell> supay: by default the wallet starts off with 2000 keys generated in it.
17:37 < achow101> that's expected. just open the file in a text editor and ctrl+f your private key
17:37 < supay> oh, wow. i see
17:38 < supay> alright, the addresses are definitely not what i expected. both have 0 balance
17:39 -!- gwillen [~gwillen@unaffiliated/gwillen] has quit [Read error: Connection reset by peer]
17:39 -!- gwillen [~gwillen@unaffiliated/gwillen] has joined #bitcoin-core-dev
17:39 < gmaxwell> are the addresses with them addresses that are in your list anywhere?
17:40 < supay> nope, they aren't! :O
17:40 -!- gwillen__ [6c5a2944@gateway/web/freenode/ip.108.90.41.68] has joined #bitcoin-core-dev
17:40 < achow101> supay: one thing that I think we forgot to mention was the address type. dumpwallet probably gives you addresses beginning with 3, yes? what do the addresses in your list begin with?
17:40 < achow101> (rescan will pick up both address types, just that exporting addresses from core will use the default type)
17:41 < supay> dumpwallet gave me 3 address corresponding with the private key. the addresses on the list begin with 1
17:41 < achow101> restart bitcoind with with -addresstype=legacy and try the dumpwallet thing again
17:42 < supay> alright, let me give that a go
17:42 < gwillen> btw supay, are you only using bitcoin core for rescuing this old wallet? I.e. you have no other funds in it?
17:42 < supay> yes, that is correct!
17:42 < gwillen> (if so, it's possible to set it up without the 2000 keys already in it, just to reduce confusion)
17:43 < achow101> gwillen: that's not in a release yet
17:43 < gwillen> o :-(
17:43 < gwillen> heh, nevermind
17:43 < achow101> would require installing 0.18.0rc2
17:45 < gmaxwell> huh? you can still set the keypool size to 1.
17:45 < achow101> oh right. I assumed gwillen was talking about blank wallets
17:46 -!- omonk [~omonk@unaffiliated/omonk] has quit [Ping timeout: 245 seconds]
17:46 < supay> achow101: restarted bitcoind with that flag. however, same addresses in dumpwallet
17:46 < gwillen> ohhh, I was, but that's clever
17:47 < gwillen> you probably need to do the import again now that the flag is set
17:47 < supay> ah, alright
17:47 < gwillen> the addresses are remembered from before, unless you make a new wallet
17:48 < gwillen> BTW, let me take this opportunity to ask: you have multiple safe copies of this list of private keys, right? Stored in at least a few places, at least one of which is not "electronically on the computer you're using right now"?
17:48 < gwillen> since you have hopefully learned your lesson about single points of failure? :-)
17:48 -!- gwillen__ [6c5a2944@gateway/web/freenode/ip.108.90.41.68] has quit [Quit: Page closed]
17:48 < supay> i don't have more copies ;((((
17:49 < gwillen> well you should make them now
17:49 < supay> hahaha, alright
17:49 < gwillen> before you do any more fiddling with the list, like, very carefully copy it to a USB stick
17:49 < gwillen> and do not even dream of getting rid of the old hard drive / backup / whatever you got it from
17:49 < gwillen> never do any more backups to that drive or touch it in any way
17:49 < gwillen> store it safely as an extra precaution
17:49 < supay> hehe, that makes sense
17:50 < supay> wait, let me retrieve the list as i had it before
17:50 < supay> i think that will shed more light to this issue
17:50 < gwillen> (for example, if there is somehow something wrong with your dump, or it is incomplete, you might have to go back to it and try to dig out more informattion)
17:50 < gwillen> in fact, consider buying another drive to make an identical copy of whatever drive that is
17:51 < achow101> supay: in your dumpwallet output, you should see multiple addresses for each private key
17:51 < gwillen> like, do the math on how much your bitcoins are worth, and consider that the bits we are helping you recover right now are effectively a pile of hundred-dollar bills of the equivalent value
17:51 < supay> gwillen: i have a few copies of the backup, on cloud. but i'll make other physical copies as well
17:51 < gwillen> and you should treat them with equivalent care
17:51 < supay> achow101: yes, that is correct. i see 3
17:51 < gwillen> okay, great
17:51 < gwillen> (hopefully encrypted in the cloud...)
17:51 < supay> yep, it is encrypted! :)
17:52 < achow101> supay: one of those should begin with a 1, that doesn't match what you have in your list?
17:53 < supay> achow101: right. there's one that begins with 1. it doesn't match :( which is why i want to get a clean dump of the files and try again. i think i messed up somewhere along the way
17:53 < achow101> do you see that address anywhere else in your list?
17:53 < supay> the one in dumpwallet? nope, that isn't present in my list
17:54 -!- omonk [~omonk@unaffiliated/omonk] has joined #bitcoin-core-dev
17:54 -!- supay [dfe622a7@gateway/web/freenode/ip.223.230.34.167] has quit [Quit: Page closed]
17:55 < achow101> I doubt that you've done anything wrong that's made this output an incorrect address. I think it's far more likely that the private keys you have simply don't match
17:56 < gwillen> oops, we lost him
17:57 -!- jtimon [~quassel@59.31.134.37.dynamic.jazztel.es] has quit [Quit: gone]
17:58 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has quit [Ping timeout: 255 seconds]
17:58 -!- zivl [~zivl@unaffiliated/zivl] has quit [Ping timeout: 245 seconds]
18:24 < gwillen> hmm, I just imported a descriptor with no private keys, and got ""All private keys are provided, outputs will be considered spendable. If this is intentional, do not specify the watchonly flag."
18:25 < gwillen> it's a sh(wsh(multi(2,A,B,C))) with fingerprints and paths given
18:25 < gwillen> and tpubs
18:26 < sipa> are the privkeys already imported?
18:26 < gwillen> although getdescriptorinfo on it returns "hasprivatekeys": false
18:26 < gwillen> shouldn't be
18:26 < sipa> in that case it sounds like there's a bug...
18:26 < gwillen> *nods*
18:27 < sipa> what does gai on one of the resulting addresses say?
18:29 < gwillen> I'm actually not sure how to get an address from it
18:29 < gwillen> that was going to be the next thing to figure out :-)
18:29 < sipa> deriveaddresses
18:29 < gwillen> oho
18:30 < gwillen> gai says ismine: false
18:31 < gwillen> gai also displays the desc in a kind of broken way, uhoh
18:31 < gwillen> it's suppose to step all three *'s together, right?
18:31 < sipa> yes
18:32 < gwillen> ahh, it looks like maybe it is, but it's forgotten the origin info for all but the first one
18:32 < gwillen> so they display without it, but they're still changing
18:32 < gwillen> this is a peak bug-filing day for me I guess :-)
18:32 < gwillen> I suppose not a lot of people are executing these codepaths yet
18:33 < sipa> yeah
18:36 < gwillen> fanquake: how do you tag those so fast :-P
18:36 < gwillen> are you a robot
18:50 -!- benthecarman [~benthecar@ics133-248.icsincorporated.com] has joined #bitcoin-core-dev
18:50 -!- drexl [~drexl@188.166.71.168] has quit [Remote host closed the connection]
18:51 -!- drexl [~drexl@188.166.71.168] has joined #bitcoin-core-dev
18:52 -!- benthecarman [~benthecar@ics133-248.icsincorporated.com] has quit [Client Quit]
18:55 -!- drexl [~drexl@188.166.71.168] has quit [Ping timeout: 250 seconds]
19:15 -!- BGL [thirty@75-149-171-58-Washington.hfc.comcastbusiness.net] has quit []
19:25 -!- zivl [~zivl@unaffiliated/zivl] has joined #bitcoin-core-dev
19:31 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
19:36 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Ping timeout: 268 seconds]
19:50 < sipa> gwillen: i have met a humanoid-looking individual who claims to be fanquake
19:54  * luke-jr wonders if the security danger to assumeutxo could be mitigated by only producing hashes encrypted to a key that controls UTXOs over a certain value
20:00 < gmaxwell> luke-jr: -ECOMPREHENSIONFAIL
20:00 < gmaxwell> forget the mechenism, what property are you trying to achieve?
20:01 < luke-jr> gmaxwell: making it infeasible to trust someone else's UTXO snapshot
20:01 < luke-jr> ie, you can generate your own on PC A, and move it to PC B, but that's it
20:02 < gmaxwell> well you can just copy your chainstate now...
20:02 < luke-jr> gmaxwell: context is a ML thread suggesting assumeutxo can only be modified in code, which not only creates a trust-the-devs problem, it also makes it useless for this legitimate use case
20:03 < gmaxwell> Personally I'm fine with "a trivially validated (root) hash is embedded in the software you run, and subject to the same review process that governs any other update to it" but I know that some people (morcos, and I think sipa) felt that was placing too much responsiblity on the developers/reviewer.  I still feel that they could be convinced otherwise, since I still haven't seen a counter to my 
20:03 < gmaxwell> primary assertion that essentially every change to the software has the same/worse security risk, but most are much harder to review.
20:03 < gmaxwell> luke-jr: which legitimate use case?
20:03 < luke-jr> gmaxwell: an easy way for users to bootstrap new nodes of their own, from an older node they already synced
20:03 < luke-jr> and/or effectively backup their chainstate
20:04 < luke-jr> (if it gets corrupted, having a commitment to a UTXO set would help recover quicker)
20:04 -!- drexl [~drexl@188.166.71.168] has joined #bitcoin-core-dev
20:05 < gmaxwell> luke-jr: I thnk that usecase is essentially irrelevant:  consider,  we must support users syncing up without that because that will always be the vast majority of the syncups,  without too much pain.   If we do, then the 'use yourself' is not much of an improvement-- you just go from acceptable to somewhat better.    Relatedly,  we must be able to catch up several months of activity in a short 
20:05 < gmaxwell> amount of time, otherwise nodes that have been offline for a couple months will not be recoverable except through a trusty process... (e.g. the case with ethereum now).
20:07 < luke-jr> I don't have a magic solution to make IBD instant.
20:07 < gmaxwell> Also, I think the problem with the kind of thinking you're offering there is that the overlap in users that want instant magic now,  can't just rsync an existing node,  and will actually go along with some complicated utxo encryption scheme is essentially zero.
20:08 < gmaxwell> luke-jr: no, but instant I think isn't really important  (and for anyone who wants faster, rsyncing a node is probably never going to be beat)
20:08 < luke-jr> it doesn't need to be complicated
20:09 < gmaxwell> luke-jr: so you have some always on AWS node .... oh now you have to take one of your valuable private keys, manually handle it, put it on it...  essentially no one will ever do that.
20:10 < gmaxwell> just rsync. rsync works fine. We can make rsync work better by havin an rpc to flush and quiesce the chainstate.
20:10 < luke-jr> it's good if it doesn't work with untrustworthy AWS nodes..
20:11 < luke-jr> rsync doesn't work for backups
20:11 < gmaxwell> What you're describing effectively doesn't work for backups either. 
20:11 < luke-jr> why not?
20:12 < luke-jr> actually, it might not even need encryption for that case: just update the wallet file with a UTXO snapshot regularly, and restore from that if it's there
20:14 < gmaxwell> Right. But all that is just begging for some 'helpful person' to fork the software and make the one line patch to remove the agreement check.  Relaying on that, instead of making sure there is little to no need for it, just makes it more likely that in practice all users end up looking like ethereum does now, pure blind trust.
20:15 -!- Eagle[TM] [~EagleTM@unaffiliated/eagletm] has joined #bitcoin-core-dev
20:15 < gmaxwell> like if you worry about corruption (A good worry), then just a chainstate recovery without any external loading would be fine.
20:15 < gmaxwell> (I believe wumpus began working on this before)
20:16 < luke-jr> for that, we need a way to freeze the chainstate db files during the backup, right?
20:16 < gmaxwell> I believe wumpus worked on a thing previously to backup the chainstate using ldb snapshots.
20:17 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has quit [Ping timeout: 250 seconds]
20:17 < gmaxwell> luke-jr: if we implement a AV utxo sync we'll get chainstate backups as a side effect most likely, since we'll need to store periodic old utxo states for people to sync from.
20:18 < gmaxwell> (and so presumably the same mechenism could optionally be run more frequently for local-only recovery at the expense of more disk space)
20:18 < gmaxwell> Though I think if people really need better than what is offered to all users then we have a problem.
20:22 < gmaxwell> I think that last point basically summarizes my view: We need to offer acceptable performance in an acceptably secure way, if we do not people will do insecure things.  If we do-- then we don't have a strong reason to implement conditionally secure local only behavior, it wouldn't be worth the effort-- because the generally available mechenism is good enough.  We also don't have some massive 
20:22 < gmaxwell> surplus of development resources (review, QA, design, development...), and we've also seen from the past the insecure half solutions (like BIP37) starve more secure secure approaches of resources.
20:37 -!- drexl [~drexl@188.166.71.168] has quit [Ping timeout: 245 seconds]
20:46 -!- BGL [eighty@75-149-171-58-Washington.hfc.comcastbusiness.net] has joined #bitcoin-core-dev
21:02 -!- shesek [~shesek@unaffiliated/shesek] has quit [Remote host closed the connection]
21:22 -!- hebasto [~hebasto@95.164.65.194] has joined #bitcoin-core-dev
21:32 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
21:36 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Ping timeout: 240 seconds]
21:54 -!- ppisati [~ppisati@net-5-95-166-237.cust.vodafonedsl.it] has quit [Quit: leaving]
22:00 -!- ppisati [~ppisati@net-5-95-166-237.cust.vodafonedsl.it] has joined #bitcoin-core-dev
22:02 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...]
22:08 -!- cornfeedhobo [~cornfeedh@unaffiliated/cornfeed] has quit [Quit: ZNC - https://znc.in]
22:14 -!- instagibbs [~instagibb@pool-100-15-135-248.washdc.fios.verizon.net] has quit [Ping timeout: 250 seconds]
22:14 -!- kljasdfvv [~flack@p200300D46F0FA500897A368518A3B6F3.dip0.t-ipconnect.de] has quit [Ping timeout: 250 seconds]
22:15 -!- kljasdfvv [~flack@p200300D46F0FA50041BE1D3480B5481D.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
22:17 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev
22:27 -!- instagibbs [~instagibb@pool-100-15-135-248.washdc.fios.verizon.net] has joined #bitcoin-core-dev
22:28 < luke-jr> gmaxwell: AV utxo sync is not acceptably secure
22:30 < gmaxwell> luke-jr: then no software is secure enough because it is strictly more secure then the rest of the software.
22:31 -!- cornfeedhobo [~cornfeedh@unaffiliated/cornfeed] has joined #bitcoin-core-dev
22:31 < gmaxwell> no compromise that can make AV fail couldn't also make the software fail, it is only protected against by review.. and yet review in general is radically harder for software in general than it is for AV.
22:33 < gmaxwell> there are a million and one subtle changes that would silently turn off validation in production, many of them would be hard for review to detect.
22:33 < gmaxwell> So how does AV make any of that worse?
22:35 -!- drexl [~drexl@188.166.71.168] has joined #bitcoin-core-dev
22:42 < luke-jr> AV utxo sync is not simple AV..
22:44 < gmaxwell> It isn't clear to me what you're saying.
22:44 < gmaxwell> What is the attack model that you are concerned with?
22:44 < luke-jr> as to your argument, it can't be strictly more secure, since it relies on the software being secure as a prerequisite to verify it
22:47 -!- benthecarman [~benthecar@ics133-248.icsincorporated.com] has joined #bitcoin-core-dev
22:47 < gmaxwell> luke-jr: It's more secure than the other vectors, not including them.  Consider a nearly unbreakable lock but which can be defeated by drilling ... in a cardboard door.  We need not debate too much about the limitation of the lock, since anyone that can drill can even more easily just drill right through the rest of the door.
22:49 < gmaxwell> if someone can sneak through a false AV hash through review and get people to run it, assuming that the software has been updated to compute AV hashes on all nodes (which is the point of pieter's rolling utxo set hash work) so that catching a false one is as trivial as just looking at the output on a node that didn't depend on the update,  ... then that person could just as well sneak through 
22:49 < gmaxwell> something to trigger bypassing scriptchecks or similar.
22:50 < gmaxwell> because it would be a lot easier to sneak a backdoor like that in, then put in a false hash that everyone could trivially check.  This is the AV security argument.
22:50 < gmaxwell> It applys no less to utxoav then it does to av in general.
22:51 < luke-jr> this does assume people *can* verify it; but it is creating a situation where it's likely people won't be able to anymore (because block sizes will get increased since it "doesn't matter" anymore)
22:54 < gmaxwell> People validate new version with the existing versions that they have.  If you have a valid version, then you can check that a new release is invalid.  So even if no one is starting from scratch you still can't get a compromise though unless you 'break the chain' of review, e.g. by having everyone who reviews leave and get replaced with new people that didn't benefit from the review of others.
22:55 < gmaxwell> and I don't think it changes the situation on block sizes much, already block sizes are well beyond what results in a sustainable initial sync, and have been for years.
22:55 < gmaxwell> sustainability of the initial sync isn't a concern there.
22:55 < gmaxwell> it's already horked.
22:55 < luke-jr> not entirely yet
22:55 < luke-jr> today, a new user *can* do the IBD
22:56 < gmaxwell> even developers of lite clients don't want to run full nodes now, come on.
22:56 < gmaxwell> luke-jr: yes, well people _can_ do an archive sync of ethereum to, which is several TB of data and months of processing on a very high end system.
22:56 < gmaxwell> s/to/too/
22:57 < gmaxwell> luke-jr: it isn't like most users can review updates except in the same kind of sense that they could buy a really high end computer system.
22:58 < gmaxwell> (in fact, _I_ can't reliably review updates, I am absolutely confident that you could slip a 'stops actually validating' backdoor past me-- but its much less likely that you could get one past everyone who looks)
22:59 < gmaxwell> (and I assume vice versa)
23:00 < gmaxwell> luke-jr: Thanks though for explaining more though. I at least agree "utxoav would be an excuse to radically crank load" is something to consider.
23:01 -!- benthecarman [~benthecar@ics133-248.icsincorporated.com] has quit [Quit: Leaving]
23:01 < gmaxwell> luke-jr: but I would suggest that the absense of this is already resulting in an outcome where very few users (even businesses) don't run full nodes, and now many that do so via highly insecure opaque snapshot installs.... and that is a lot worse.
23:02 < luke-jr> gmaxwell: the ability to review code is a boolean; the ability to review a UTXO AV grows with IBD time
23:02 < luke-jr> although I suppose the "an excuse to radically crank load" problem exists with *any* solution in this area
23:02 < gmaxwell> luke-jr: not quite. AV review effort is a constant if you've already got a node running.
23:03 < luke-jr> gmaxwell: new users shouldn't be forced to trust existing users in this way
23:03 < gmaxwell> but thats also what happens with software review, reviewing the whole of the software is well beyond the capabilities of any single person.
23:03 < gmaxwell> at best we can hope people review updates.
23:04 < luke-jr> reviewing code is soemthing new users *can* do, even if not as a single person
23:04 < gmaxwell> luke-jr: and 'forced' is too strong, you can sync history, even if its costly to the point of being obnoxious: it already is and the result is people not running full nodes, in vast numbers.
23:05 < luke-jr> that can change if we get 1 TB blocks
23:05 < gmaxwell> no, because "1 tb blocks" (or similar jokes) isn't remotely viable for continued operation.
23:05 < gmaxwell> luke-jr: even if capacity were cranked up 'new users' as a group could do a sync from nothing without AV... even if it was so expensive as to make it unrealistic for just a single user to do alone.
23:06 < gmaxwell> and any load level that was viable for single users to keep _running_ would remain viable for a group of users to history validate for a long time.
23:07 -!- drexl [~drexl@188.166.71.168] has quit [Ping timeout: 250 seconds]
23:12 < gmaxwell> as an aside, I really don't think that handicapping the software is a viable way to avoid load increases that compromise security.
23:12 < gmaxwell> All that approach would do in the long run is demand someone forks the software, fixes the shortcoming, and then pushes you out of the way-- killing your influence.
23:16 -!- Skirmant [~Skirmant@78-62-14-181.static.zebra.lt] has quit [Ping timeout: 268 seconds]
23:17 < gmaxwell> it would be like if you had applied this same logic to compact blocks and  managed to block their introduction... BU would have eventually gotten 'xthin' working right, and then essentially all miners would use it, and many nodes (due to the bandwidth savings).
23:18 -!- Eagle[TM] [~EagleTM@unaffiliated/eagletm] has quit [Ping timeout: 246 seconds]
23:18 < luke-jr> compact blocks doesn't have this problem, though?
23:18 < luke-jr> hmm, I guess in a sense it could be argued to
23:19 < gmaxwell> sure it could easily be argued to
23:19 < gmaxwell> (and in fact many people argued that blocksize was no longer an issue because of compact blocks, but most of those people were confused about what compact blocks did)
23:20 < gmaxwell> there have been a bunch of threads saying essentially "compact blocks made blocks 99% smaller so blocks can now be 100x larger!"
23:24 -!- Skirmant [~Skirmant@78-62-14-181.static.zebra.lt] has joined #bitcoin-core-dev
23:26 < gmaxwell> but my point is that to the extent people thinking that is a risk, you can't solve it by just not making the improvement. Someone else will make it, failing to make it would only just diminish your relevance for future changes.
23:26 < gmaxwell> (and not only that, it would make true some of the absurd allegations of unethical behavior, e.g. intentionally degrading the system)
23:27 < gmaxwell> (or at least arguably make them true)
23:30 < luke-jr> could have different priorities until someone else starts working on such features
23:31 < luke-jr> not that I'm saying it's necessarily a solution, just a possible alternative to "don't do it and lose users"
23:32 < gmaxwell> Right, but even with that it still points out that 'don't implement a performance improvement' isn't really a way to effectively avoid security loss in the long run.
23:33 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has joined #bitcoin-core-dev
23:33 < gmaxwell> and 'people will switch software' is only one risk factor.  For something like initial sync, what happens is that many people don't run fullnodes at all: and we've seen that happen and those folks are overwhelming more likly to think "wtf is capacity limited at all!".
23:34 < gmaxwell> the analog for compact blocks is e.g. miners doing 'headers only mining' then not caring about propagation/validation speed.
23:35 < gmaxwell> which, indeed, had become very common before compact blocks
23:35 < gmaxwell> (and which was a material driver for the development and deployment of compact blocks,  ... not alternative software though that would have also been one had your approach been taken there)
23:36 < gmaxwell> In any case, point is that ignoring a need isn't very effective-- it'll be routed around in some way, and the routing around may have much worse outcomes.
23:37 -!- jarthur [~jarthur@2605:6000:1019:41ab:78fa:e73d:546f:f400] has quit [Ping timeout: 258 seconds]
23:40 < gmaxwell> (of course the degree depends on how serious the 'need' is...)
23:41 < gmaxwell> (and how easy the bypasses are, unfortuantely "trust someone else" is almost always a very easy bypass that a very large number of economically important users are happy to make)
23:50 -!- mn9495885 [~nodebot@cpe-67-243-195-224.nyc.res.rr.com] has quit [Ping timeout: 250 seconds]
23:50 -!- mn94958863 [~nodebot@cpe-67-243-195-224.nyc.res.rr.com] has quit [Ping timeout: 250 seconds]
23:50 -!- mn949588 [~nodebot@cpe-67-243-195-224.nyc.res.rr.com] has quit [Ping timeout: 250 seconds]
23:50 -!- mn9495881 [~nodebot@cpe-67-243-195-224.nyc.res.rr.com] has quit [Ping timeout: 245 seconds]
--- Log closed Thu Apr 04 00:00:35 2019