--- Log opened Sat Sep 05 00:00:11 2020 00:27 -!- reallll [~belcher@unaffiliated/belcher] has joined #joinmarket 00:30 -!- belcher_ [~belcher@unaffiliated/belcher] has quit [Ping timeout: 240 seconds] 00:35 -!- reallll is now known as belcher 01:11 < waxwing> ah interesting. sounds like "fast" sync (no --recoversync) is failing. it's definitely not unconfirmed-ness causing that, since wallet-tool display methods always include unconfirmed. 01:11 < waxwing> cbeams, ^ 01:17 -!- jonatack [~jon@37.166.58.175] has joined #joinmarket 01:23 -!- cbeams [~cbeams@unaffiliated/cbeams] has quit [Remote host closed the connection] 01:24 -!- Evanito [~Evanito@cpe-76-87-174-228.socal.res.rr.com] has quit [Read error: Connection reset by peer] 01:25 -!- cbeams [~cbeams@unaffiliated/cbeams] has joined #joinmarket 01:25 < cbeams> waxwing: I've tracked it down to a single address that is missing when --recoversync is not enabled 01:26 < cbeams> and that address has just one transaction against it with hundreds of confirmations 01:27 < waxwing> that's a different error from what you were reporting at the start (originally you were reporting only that 'history' was showing a mismatch). the question is, what changed around the time that `wallet-tool` default (display) started showing the wrong balance? 01:28 < waxwing> i'm trying to remember others, but the only scenario i know where fast sync (default) fails is where you are on a new Bitcoin Core instance and it doesn't have the history in the Core wallet (need rescan etc.) 01:28 < waxwing> if there are others, it's important we know about them. 01:38 < waxwing> it's interesting, if i read right, you report `listtransactions` returning an empty array .. what i'm wondering now is if this is somehow to do with a wallet setting in Core. 01:38 < waxwing> like did you at some point set `rpc_wallet_file` in the joinmarket config, or did you start running Core itself with a new wallet. 01:40 < waxwing> because it seems to me if `listtransactions` returns nothing, there is no possible way that you can get a correct balance report (except with -no-history which checks the utxo set) 01:41 < cbeams> yes, I'm running bitcoin-rpc-no-history in conjunction with recoversync 01:41 < cbeams> will be back later to follow up further, thanks. 01:41 < waxwing> yes the no-history is basically guaranteed to work as belcher says. 01:41 < waxwing> but why listtransactions is returning nothing is the real issue here. 02:00 < belcher> listtransactions return nothing if you dont pass any arguments 02:00 < belcher> you need to pass something like `listtransactions * 0 10 true` 02:09 -!- asy [~asymptoti@gateway/tor-sasl/asymptotically] has joined #joinmarket 02:10 -!- rdymac [uid31665@gateway/web/irccloud.com/x-yzzsmwznkkvtuhop] has joined #joinmarket 02:12 < waxwing> oh yes, rather good point :) 02:45 -!- vrana [~mvranic@gateway/tor-sasl/vrana] has quit [Ping timeout: 240 seconds] 02:46 -!- jonatack [~jon@37.166.58.175] has quit [Read error: Connection reset by peer] 02:51 -!- vrana [~mvranic@gateway/tor-sasl/vrana] has joined #joinmarket 03:06 < cbeams> belcher waxwing: I do see the offending transaction in the results from `listtransactions "*" 20 100 true`. Reports the same as my block explorer running against my node: that there are hundreds of confirmations for that tx. 03:06 < belcher> whats different about the offending transactions? how do you know that tx is special 03:07 < belcher> cbeams ^ 03:07 < cbeams> I narrowed it down to that transaction by doing the following: 03:08 < cbeams> 1. comparing the output of `summary` with no-history and --recoversync to `summary` run without these options, and saw that only mixdepth 1 had a different balance. All other mixdepths (I have 6) had the same balance. 03:09 < cbeams> 2. From there, I compared mixdepth 1 output from `display` with no-history and --recoversync to `display` run without these options 03:09 < cbeams> and saw that there were two derivation paths present in the former that were missing in the latter. 03:09 < cbeams> I'll paste a scrubbed output, one moment. 03:16 < cbeams> Excerpted from the output of `python scripts/wallet-tool.py wallet.jmdat display` with no special options: 03:16 < cbeams> m/49'/0'/1'/1/130 3... 0.04200000 change-out 03:16 < cbeams> m/49'/0'/1'/1/131 3... 0.00420000 change-out 03:16 < cbeams> m/49'/0'/1'/1/134 3... 0.00042000 change-out 03:16 < cbeams> m/49'/0'/1'/1/135 3... 0.00004200 change-out 03:16 < cbeams> note that /132 and /133 are missing here, and that addresses are listed as 'change-out'. 03:16 < cbeams> Excerpted from the output of `python scripts/wallet-tool.py wallet.jmdat display --recoversync` with the bitcoin-rpc-no-history option enabled in joinmarket.cfg: 03:16 < cbeams> m/49'/0'/1'/1/130 3... 0.04200000 used 03:16 < cbeams> m/49'/0'/1'/1/131 3... 0.00420000 used 03:16 < cbeams> m/49'/0'/1'/1/133 3... 0.01112222 used 03:16 < cbeams> m/49'/0'/1'/1/134 3... 0.00042000 used 03:16 < cbeams> m/49'/0'/1'/1/135 3... 0.00004200 used 03:16 < cbeams> note that while /132 is still missing, /133 is now present. The amount on that address accounts for the total discrepancy in my current balance vs. my historical balance. Also note that all addresses are now listed as 'used'. 03:18 -!- Antwon8Breitenbe [~Antwon8Br@static.57.1.216.95.clients.your-server.de] has joined #joinmarket 03:20 < cbeams> 3. From there, I just looked at the offending address (/133) in an explorer and saw that it has a single transaction against it, responsible for the whole amount on the address, and that this transaction was confirmed days ago, with 800+ confirmations. I see that same transaction in the output of `listtransactions` as mentioned above. So the question is, why does joinmarket not recognize this tx / utxo next to all the other, similarly 03:20 < cbeams> long-ago confirmed transactions? 03:22 -!- cbeams_ [~cbeams@unaffiliated/cbeams] has joined #joinmarket 03:25 -!- cbeams [~cbeams@unaffiliated/cbeams] has quit [Ping timeout: 260 seconds] 03:28 < cbeams_> test 03:31 < cbeams_> Another clue in all of this is that I noticed that `history` wasn't reporting the complete list of transactions quite a while (months) ago, but I never looked into it more deeply because there were no other anomalies, i.e. my balance was fine. This balance discrepancy problem showed up somewhere between Aug 25th and yesterday (Sept 4th). Aug 25th was the last time I checked in on my yield generator and ran `summary`, etc, and everythi 03:31 < cbeams_> ng was fine then. I had updated to bitcoin core 0.20.1 a couple days before that, and I had to restart the node several times in the days following. So it seems possible that it was something in that upgrade or those restarts that is responsible here. 03:31 < cbeams_> belcher: ^ 03:37 < belcher> sounds like that for some reason joinmarket missed that one transaction 03:43 < waxwing> at this point a complete guess, but i'm wondering if a conflict (double spend or reorg) might be involved. 03:43 < waxwing> thanks for the detailed report cbeams_ 03:44 < waxwing> one detail that could matter: is that one discrepancy transaction a multi-party coinjoin (JM style), or a deposit of some kind? 03:53 < waxwing> cbeams_, if you have a chance could you try the following : `bitcoin-cli listaddressgroupings` and then grep the result (it's big so put it in a file) for the address at index /132 03:54 < waxwing> i presume it won't be there but that will at least pinpoint the failure 03:55 < waxwing> (to be clear, you probably need: `bitcoin-cli -rpcwallet=walletname listaddressgroupings` 03:58 < waxwing> oh that line of thought gave me an idea about a corner case that could cause this. investigating. 04:15 < waxwing> ok did some experiments manually and i think i confirmed the hypothesis: 04:15 < waxwing> if you do a sweep transaction to another address in the wallet, it will override the wallet label with a new label "joinmarket-notify", and that address will no longer be reported as belonging to the wallet (so funds shown missing). 04:16 < belcher> ooooo, good find 04:16 < waxwing> anyone inclined to take a look at this, see: https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/jmclient/jmclient/wallet_service.py#L216-L224 04:17 < waxwing> by 'manual experiment', i mean, i was just checking that if you `importaddress/multi` with one label and then do it again with a different label, it does indeed overwrite without complaint. 04:17 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has joined #joinmarket 04:18 < waxwing> so if that's correct cbeams_ you would find that `getaddressesbylabel` rpc with argument of "joinmarket-notify" will show that /132 address. 04:18 < waxwing> and if so this is a bug that needs fixing, although it's a little tricky. 04:19 < waxwing> oh; no it isn't tricky. just check if the address belongs to the wallet before putting the *notify label. 04:20 < waxwing> huh, maybe i'm wrong, or maybe 'is_address_imported' isn't working right. 07:33 < waxwing> cbeams_, what was the version you were using before 0.20.0 of bitcoin core and (if you remember) did you have `-deprecatedrpc=accounts` specified? 07:34 < waxwing> (cutting long story short, the above reasoning is probably correct if you were running an old version that supported the getaccount RPC) 07:56 -!- waxwing [~waxwing@unaffiliated/waxwing] has quit [Ping timeout: 246 seconds] 08:02 -!- Evanito [~Evanito@cpe-76-87-174-228.socal.res.rr.com] has joined #joinmarket 08:35 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has quit [Quit: Leaving] 08:47 -!- tralfaz [~dulyNoded@94.198.42.121] has quit [Ping timeout: 264 seconds] 08:48 -!- tralfaz [~dulyNoded@69.4.234.77] has joined #joinmarket 09:01 -!- tralfaz [~dulyNoded@69.4.234.77] has quit [Remote host closed the connection] 09:03 -!- tralfaz [~dulyNoded@69.4.234.77] has joined #joinmarket 09:04 -!- tralfaz [~dulyNoded@69.4.234.77] has quit [Remote host closed the connection] 09:04 -!- tralfaz [~dulyNoded@69.4.234.77] has joined #joinmarket 09:09 -!- tralfaz [~dulyNoded@69.4.234.77] has quit [Remote host closed the connection] 09:23 < cbeams_> belcher waxwing: just seeing the above now, thanks. Will get back with responses. 09:53 -!- tralfaz [~dulyNoded@69.4.234.77] has joined #joinmarket 10:02 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has joined #joinmarket 11:45 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has quit [Remote host closed the connection] 11:45 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has joined #joinmarket 12:09 -!- rdymac [uid31665@gateway/web/irccloud.com/x-yzzsmwznkkvtuhop] has quit [Quit: Connection closed for inactivity] 14:10 -!- rdymac [uid31665@gateway/web/irccloud.com/x-kyubrvrondtdyucn] has joined #joinmarket 14:14 -!- cbeams_ [~cbeams@unaffiliated/cbeams] has quit [Remote host closed the connection] 14:17 -!- cbeams [~cbeams@unaffiliated/cbeams] has joined #joinmarket 15:53 -!- tralfaz [~dulyNoded@69.4.234.77] has quit [Ping timeout: 264 seconds] 15:53 -!- tralfaz [~dulyNoded@69.4.234.112] has joined #joinmarket 16:05 -!- asy [~asymptoti@gateway/tor-sasl/asymptotically] has quit [Quit: Leaving] 17:08 -!- DSRelBot [~DSRelBot@p54866f69.dip0.t-ipconnect.de] has quit [Ping timeout: 240 seconds] 17:08 -!- HackRelay [~jmrelayha@p54866f69.dip0.t-ipconnect.de] has quit [Ping timeout: 265 seconds] 17:21 -!- HackRelay [~jmrelayha@p5de4a114.dip0.t-ipconnect.de] has joined #joinmarket 17:22 -!- DSRelBot [~DSRelBot@p5de4a114.dip0.t-ipconnect.de] has joined #joinmarket 18:17 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has quit [Remote host closed the connection] 20:09 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has quit [Remote host closed the connection] 20:09 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection] 20:10 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has joined #joinmarket 20:10 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has joined #joinmarket 20:58 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has joined #joinmarket 21:24 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has joined #joinmarket 21:24 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection] 21:41 -!- tralfaz [~dulyNoded@69.4.234.112] has quit [Ping timeout: 264 seconds] 21:41 -!- tralfaz [~dulyNoded@69.4.234.91] has joined #joinmarket 21:41 -!- tralfaz [~dulyNoded@69.4.234.91] has quit [Remote host closed the connection] 22:03 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has joined #joinmarket 22:05 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection] 22:35 -!- kristapsk_ [~KK@gateway/tor-sasl/kristapsk] has joined #joinmarket 22:35 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has quit [Ping timeout: 240 seconds] --- Log closed Sun Sep 06 00:00:12 2020