--- Day changed Mon Dec 09 2019
00:55 -!- technonerd [~techno@gateway/tor-sasl/technonerd] has quit [Ping timeout: 260 seconds]     
01:14 -!- technonerd [~techno@gateway/tor-sasl/technonerd] has joined #joinmarket     
02:07 -!- avril [wha@unaffiliated/avril] has quit [Ping timeout: 276 seconds]     
02:38 -!- avril [wha@cute.b.oy.hu] has joined #joinmarket     
02:38 -!- avril [wha@cute.b.oy.hu] has quit [Changing host]     
02:38 -!- avril [wha@unaffiliated/avril] has joined #joinmarket     
03:04 -!- Julien26Schmitt [~Julien26S@ns334669.ip-5-196-64.eu] has joined #joinmarket     
03:15 -!- reallll [~belcher@unaffiliated/belcher] has joined #joinmarket     
03:16 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 265 seconds]     
03:19 -!- oldseed [b9d4aa77@185.212.170.119] has quit [Ping timeout: 260 seconds]     
03:59 < waxwing> see #470 in relation to onion_kirens question above (and #469, same thing); this is basically pretty urgent since the bug will have people depositing coins and then not seeing them, no money loss danger but seriously inconvenient and bothersome for users who deposit more than a few times to a wallet with the latest version of code.
04:00 < waxwing> if you want the backstory on how this bug arose it's pretty complicated but short version is fast sync didn't handle deposits correctly, but mostly people didn't notice as they weren't using it much, also the "caching imports" thing i did in 0.5.5 that was then reverted due to performance issues, hid the problem.
04:01 < waxwing> "weren't using it much" - it = fast sync
04:01 < waxwing> anyway we will have to issue a 0.6.1 imo because of this - it's a broad enough and annoying enough bug for that.
04:17 < AgoraRelay> [agora-irc/AlexCato] I can help test this. How do I reproduce the problem? My understanding is: use version 0.6: use "wallet-tool.py <walletfile>" --> deposit to 5th external address -> use wallet-tool again to get new external addresses displayed --> deposit to 6th external address --> this last one will not show up?
04:31 < waxwing> AlexCato thanks, yes, that's how i tested it
04:31 < waxwing> it was reproducible on regtest and #470 fixed it in my tests
04:31 < waxwing> sorry i should have probably made that clearer
04:32 < waxwing> there is a tradeoff here that the sync is slowed down by either 1-2 seconds (SSD) or more on HDD *if* you don't have the fix to Bitcoin Core importmulti (I think the fix is in 0.19 ? can someone check?)
04:33 < waxwing> but that seems both necessary and acceptable; the reversion of the previous caching was because it was happening during coinjoins; this is only for sync.
05:33 -!- slivera [slivera@gateway/vpn/privateinternetaccess/slivera] has quit [Remote host closed the connection]     
05:50 -!- reallll is now known as belcher     
05:55 < AgoraRelay> [agora-irc/AlexCato] ... blocks are coming slow, tests still need a while (need at least 4 onchain TX to reproduce, then verify fix ;( ) ...
06:05 < belcher> the Core PR which fixes the import slowness is https://github.com/bitcoin/bitcoin/pull/15741 so it looks to me like it is inside 0.19
06:12 < belcher> im testeing it but its pretty clear that the fix is in 0.19
06:13 < belcher> 0.19 imported 10k addresses in 14 seconds, iv run the same command in 0.18 and it hasnt returned yet after a couple of minutes
06:14 < AgoraRelay> [agora-irc/AlexCato] tried to reproduce the error in #470 as decribed above. Didn't work, I can see the coins (first deposited into external address #5, then next in #9). Does it have to be a new wallet for some reason? Tried it with an existing one  @waxwing
06:20 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Remote host closed the connection]     
06:24 < belcher> ah it finished, 15minutes for 0.18 and 14 seconds for 0.19
06:24 < belcher> to import 10k addresses
06:27 < waxwing> AlexCato yes it would have to be a new wallet. see the comment about the reversion of a commit that was caching imports
06:27 < waxwing> if you use a wallet created before 0.6 you wouldn't see this effect (not 100% for sure but very likely)
06:29 < waxwing> it's a complicated situation, to say the least; it will also be affected by how often you did a detailed sync, which will depend on when you switched over to 0.6 (or recent commit post #359) etc etc
06:29 < waxwing> but it is clearly reproducible with a new wallet in regtest and i have 2 reports (may be same person) that it happened to them on mainnet
06:30 < waxwing> and also the logic is pretty clear; look at how fast sync (get_address_usages) works and consider what happens if someone only ever syncs via that
06:30 < waxwing> belcher, thanks for that
06:47 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
06:49 < waxwing> alexcato belcher fwiw i'll do a test on mainnet too, and try to kind of 'stress' it and make sure a new wallet behaves exactly as expected in terms of which addresses can be succcessfully deposited too.
06:49 < waxwing> might take an hour or two here or there but nbd it's not like i have to sit and wait for it :)
06:49 < belcher> ok ty
06:50 < belcher> its interesting what you've found about importing an address with a transaction already known to core, which makes the imported address already be populated without a rescan
06:51 < belcher> there is the rpc call `importprunedfunds` which is another way to add a transaction to the wallet that doesnt need a rescan, its interesting to note you dont need to specify which imported address is associated with that transaction... all you need to specify is the transaction itself
06:51 < belcher> so that fits with what you've found
06:55 < waxwing> yeah i checked on #bitcoin the other day and ghost43 confirmed he saw the same behaviour
07:04 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Remote host closed the connection]     
07:09 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
07:13 < waxwing> ok test already done (i realised there is no need to wait for confs to see it, so it was fast)
07:15 < waxwing> deposited in index 3 and index 5 of mixdepth 2 after initial wallet create. then reloaded (did in Qt actually), saw correct display of : index 3,5,6,7,8,9,10,11. then deposited to 9 and 11 (chose 11 obviously because it's the highest index that should at that point be imported).
07:15 < waxwing> all deposits were from external wallet (electrum) to avoid interference of the type we described above re: Core import weirdness.
07:16 < waxwing> i realise now that i only tested the success case not the failure. not sure if i can be bothered with the latter, everything seems very clear at this point.
07:31 < AgoraRelay> [agora-irc/AlexCato] @waxwing: kids woke up, wont be able to finish the tests right now. Later tonight, I hope
07:35 < waxwing> ok no worries
11:09 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has joined #joinmarket     
11:11 < kristapsk> about #469 - bug does not hit if you use qt and restart after deposit to 5th address (you need to, as otherwise 6th address to deposit does not appear in UI)
11:12 < onionkirens> @waxwing: question about the issue I was getting a couple days ago with not being able to get joinmarket-qt or wallet-tool to RPC addaddress a deposit beyond index 6: is there some way that I'm using joinmarket that deviates from other others? It can't be the case that so many users are unable to see deposits beyond their 6th index at mixdepth 0. 
11:12 < kristapsk> ahh, no, it does - 6th deposit appears if you have qt running when it happens, but it disappears after qt restart
11:13 < onionkirens> interesting. I'll restart qt and try again
11:14 < onionkirens> oh damn, I just read that a little more carefully
11:15 < kristapsk> yeah, it will not help, patch / v0.6.1 will be needed, I'm just testing it various ways now
11:15 < onionkirens> Okay, I am reading the issue now and will follow workaround
11:16 < kristapsk> also note that rescanblockchain workaround only works after deposit tx is confirmed on the blockchain
11:17 < belcher> that makes scan, im pretty sure rescans dont check the mempool
11:17 < onionkirens> cool thanks. Yes it's deep in the chain by this point
11:17 < belcher> that makes sense*
11:19 < kristapsk> another minor issue maybe is that both before and after fix, if you deposit to m/49'/x/x/0/5 in the GUI, 6th address will not automatically appear until you restart
11:19 < kristapsk> but that's not directly related I think, should be something to change in the GUI in future maybe
11:20 -!- Julien26Schmitt [~Julien26S@ns334669.ip-5-196-64.eu] has quit [Ping timeout: 268 seconds]     
11:21 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has quit [Remote host closed the connection]     
11:32 < waxwing> kristapsk, yes i also realised that deposits won't affect which addresses are displayed, but that is a minor point,we agree on that (at least compared to the bug)
11:33 < waxwing> kristapsk, i didn't understand " 6th deposit appears if you have qt running when it happens, but it disappears after qt restart" .. could you elaborate?
11:33 < kristapsk> waxwing, that was before your fix, on current master - while qt is running it appears, but after restart, it's gone, 0.00000000 BTC there
11:34 < kristapsk> but #470 fixes that
11:34 < waxwing> kristapsk, ah. interesting. i can see why that happens *i think* .. it's pretty obscure.
11:34 < waxwing> but yes academic considering we are fixing the bug.
11:35 < waxwing> it might depend where you sourced the coins from, i.e. if from another joinmarket wallet on the same machine, or directly from the bitcoin core wallet on that machine.
11:35 < waxwing> see the weirdness about imports i mentioned there.
11:37 < kristapsk> it was from Bitcoin Core wallet on the same machine (where also JM addresses are as watch-only)
11:39 < waxwing> is "6h deposit" /0/5 or /0/6 ? and this is a new JM wallet right?
11:42 < kristapsk> waxwing, I made two deposits, one to 0/5, then restart qt and then to 0/6
11:47 < waxwing> kristapsk, right. i get it now, it's if anything because our code is too powerful :)
11:47 < waxwing> see there's a difference between what's cached in the wallet as a set of addresses/scripts and what's imported.
11:47 < waxwing> when the transaction_monitor loop sees a new transaction, it calls process_new_tx() which calls add_new_utxos() (in the Wallet object)
11:48 < waxwing> i made it work like that to be completely general; when the tx doesn't relate to anything in the wallet it's just a no-op
11:48 < waxwing> but here we have a situation where the transaction shows up because of your Core wallet import (if it was an external deposit it wouldn't get returned by listtransactions)
11:49 < waxwing> or your Core wallet address say (the funding), not 'import' per se
11:49 < waxwing> so it gets processed by that monitoring loop and seen because the *addresses* in the gap (next 6) are stored in the script_map object of the wallet.
11:51 < waxwing> but when you restart you call `sync_unspent()` which uses rpc listunspent and then *filters on our wallet label*, and since we didn't import, it wasn't picked up.
11:52 < waxwing> i know this is all stupid complicated, but as i said the other day, the root problem was not fixing to the rule that we "never display unless imported". once we do, we know we are "safe" that users never do stuff that leads to this kind of weirdness of unmonitored addresses or transactions in any situation.
11:53 < waxwing> and the reason this rule was not followed was ... well it was a bug from a long time back as far as i can tell, but a bunch of other behaviours meant the bug never showed itself.
11:56 < waxwing> ok so we have 2 tACKs(including me) and one utACK so i'll probably just merge it. any objections?
12:31 < kristapsk> waxwing, no objections, but AlexCato said above he hope to test this later tonight, so maybe wait till tomorrow?
12:52 < waxwing> yep kristapsk 
14:14 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has joined #joinmarket     
14:15 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has quit [Remote host closed the connection]     
14:26 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Read error: Connection reset by peer]     
14:33 < AgoraRelay> [agora-irc/AlexCato] could reproduce the problem with a new wallet on main net with 0.6.0. Will now try the same thing with 0.6.1.
14:34 < AgoraRelay> [agora-irc/AlexCato] i did *not* need a bitcoin core rescan to make the funds visible in 0.6.0. The "--recoversync" in wallet tool alone was enough
14:42 < waxwing> AlexCato did you see the backscroll about that?
14:42 < waxwing> <waxwing> it might depend where you sourced the coins from, i.e. if from another joinmarket wallet on the same machine, or directly from the bitcoin core wallet on that machine.
14:42 < waxwing> <waxwing> see the weirdness about imports i mentioned there.
14:43 < AgoraRelay> [agora-irc/AlexCato] ah, i missed that
14:43 < waxwing> it's also mentioned in the PR; this is an obscure thing, but it screws up testing
14:43 < AgoraRelay> [agora-irc/AlexCato] successfully tested 0.6.1 as well now: fix works. Will comment in github shortly
14:44 < waxwing> well 'in the PR' it's like the second comment or something
14:57 -!- technonerd [~techno@gateway/tor-sasl/technonerd] has quit [Remote host closed the connection]     
14:58 -!- technonerd [~techno@gateway/tor-sasl/technonerd] has joined #joinmarket     
15:46 -!- Zenton [~user@unaffiliated/vicenteh] has quit [Ping timeout: 265 seconds]     
15:55 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
16:01 -!- slivera [slivera@gateway/vpn/privateinternetaccess/slivera] has joined #joinmarket     
16:05 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has joined #joinmarket     
16:35 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has quit [Quit: Leaving]     
16:43 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 245 seconds]     
16:44 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
17:09 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has joined #joinmarket     
17:11 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has quit [Client Quit]     
17:11 -!- onionkirens [~onionkire@gateway/tor-sasl/onionkirens] has joined #joinmarket     
17:22 -!- AgoraRelay [~jmrelayfn@p5DE4A9D5.dip0.t-ipconnect.de] has quit [Ping timeout: 250 seconds]     
17:24 -!- CgRelayBot [~CgRelayBo@p5DE4A9D5.dip0.t-ipconnect.de] has quit [Ping timeout: 268 seconds]     
17:32 -!- AgoraRelay [~jmrelayfn@p5DE4AA01.dip0.t-ipconnect.de] has joined #joinmarket     
17:34 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has quit [Remote host closed the connection]     
17:34 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has joined #joinmarket     
17:36 -!- CgRelayBot [~CgRelayBo@p5DE4AA01.dip0.t-ipconnect.de] has joined #joinmarket     
17:43 < kristapsk> waxwing, I kinda understand that 0.6.1 should be bugfix-only release, but maybe it's worth merging #466 also in? just that it could help in a support for new users...
18:55 -!- MaxSan [~four@37.120.210.211] has joined #joinmarket     
20:42 -!- MaxSan [~four@37.120.210.211] has quit [Quit: Leaving.]     
22:24 -!- AgoraRelay [~jmrelayfn@p5DE4AA01.dip0.t-ipconnect.de] has quit [Ping timeout: 265 seconds]     
22:25 -!- CgRelayBot [~CgRelayBo@p5DE4AA01.dip0.t-ipconnect.de] has quit [Ping timeout: 250 seconds]     
22:36 -!- AgoraRelay [~jmrelayfn@p54866B32.dip0.t-ipconnect.de] has joined #joinmarket     
22:42 -!- CgRelayBot [~CgRelayBo@p54866B32.dip0.t-ipconnect.de] has joined #joinmarket