--- Day changed Sat Nov 12 2016
00:02 < GithubBot5678> [joinmarket] WyseNynja opened pull request #654: Fix install deps (develop...patch-1) https://git.io/vXKuP
00:27 -!- windsok [~windsok@45.63.59.8] has quit [Ping timeout: 244 seconds]     
00:32 -!- windsok [~windsok@45.63.59.8] has joined #joinmarket     
01:02 -!- windsok [~windsok@45.63.59.8] has quit [Ping timeout: 260 seconds]     
01:26 -!- coins123 [~coins123@unaffiliated/coins123] has joined #joinmarket     
01:59 -!- windsok [~windsok@45.63.59.8] has joined #joinmarket     
03:44 < belcher> --fast might need to be renamed, since its a verb with no noun
03:44 < belcher> tumbler.py --fast looks like it might make your tumbling faster or something
03:44 < belcher> --fast-sync instead?
03:44 < btcdrak> yes, that's a point.. but you should really think about making --fast-syn the default
03:45 < btcdrak> and add another switch for --deep/legacy/slow-sync or whatever you want to call it.
03:47 < belcher> --fast-sync only works if you're already sync'd
03:48 < btcdrak> if you arent synced and you use that, does it spew an error?
03:48 < belcher> hmm     
03:48 < belcher> i didnt write it but i think it might not be possible to tell
03:48 < belcher> but if it is, that would be good
03:48 < belcher> sec, looking at the code
03:49 < btcdrak> if you can detect it just do an original sync (spit out a message, "performing initial sync, will take a while..."
03:56 < belcher> looking at the code, it could be detected i think if you check whether wallet.index_cache is not all zeros, and/or whether wallet_name is an existing label in bitcoin core, but ill let waxwing confirm since he wrote the code
03:56 < belcher> the commit that added it is here btw https://github.com/JoinMarket-Org/joinmarket/commit/0281900e5bfbd2720ccae6b7546e6bad303d28da
04:01 <@waxwing> belcher: yes. basically, as i mentioned in the release notes, if it's not already synced such that there isn't already an index_cache, then --fast can't work.
04:02 <@waxwing> if there *is* an index cache, great, but fast is assuming it's up to date.
04:02 <@waxwing> if it's not, you'll get an incorrect balance.
04:02 <@waxwing> the whole thing is very tricky not least because of the various scenarios: recovering a wallet, a new Core instance, an index_cache which is not accurate, whether the gap limit is or is not sufficient.
04:03 <@waxwing> and as we mentioned on the issue at the time (so saying for btcdrak 's benefit), the reason i applied --fast as an explicit option is so that an unsuspecting user doesn't have it being used, reporting incorrect balance, and not knowing why.
04:03 < belcher> hmm yes, so a wrong index_cache might happen if you have the same wallet on two separate instances
04:04 <@waxwing> the history of this is pretty complicated. belcher 's original code worked fine by not assuming anything, and building up an accurate sync from a series of steps. but then we started to get more and more spam requests from spy/spies.
04:05 <@waxwing> started to get complaints of "im having to run sync over and over" and we started hacking around addr_req_count.
04:05 <@waxwing> so i "fixed" that by using index_cache where possible to deduce how many imports were needed, but on reflection it seems it wasn't a good solution (albeit it did solve the endless re-run sync problem)
04:06 < belcher> as well as spies, it was also just slow
04:06 <@waxwing> then someone else pointed out you can query history with getaddressusages, and so --fast allows a person with a big wallet to do the whole thing cleanly, *assuming* index_cache is up to date.
04:07 <@waxwing> i feel like it would be good to rewrite the whole thing from scratch .. it should be possible for it to be cleaner, *although* don't assume it'll necessarily be *really* simple, because this kind of two-layer 'watch only address and then JM HD wallet on top' thing, plus the fact that we inevitably use tons of addresses, makes it innately at least somewhat complex.
04:09 <@waxwing> (minor point but i often wondered if index/index_cache maybe was not necessary to have as two variables; i.e. you could just treat the data in file as the persistence/cache of index)
04:09 <@waxwing> but even if so that doesn't matter really, that's not what makes it complicated.
04:10 < belcher> right so at the start the code just does index = index_cache
04:11 <@waxwing> if that's the case, it's more just about readability i guess
04:11 < belcher> i cant really think of any situations where index_cache would be wrong, except where someone is running the same wallet twice on different instances
04:12 <@waxwing> yeah i guess so, it's really about people switching Core instances, or also i guess they could make mistakes (like use an earlier backup)?
04:12 <@waxwing> well, maybe that's not even a mistake, if you lost the most up to date wallet.json, it's not a mistake to use an earlier one i think :)
04:12 < belcher> ok     
04:13 <@waxwing> there was also a confusion about what happens with tumbler, because index_cache length gets changed by it.
04:13 < belcher> wallet.json isnt really a file you back up, your backup is the seed
04:13 <@waxwing> actually i was going to talk to you about that
04:13 <@waxwing>  yes i know but people do that kind of thing
04:13 < belcher> yes     
04:18 < belcher> another thing is id like an irc logging bot here, so the stuff we talk about is recorded and able to be referenced
04:18 < belcher> i put in a request to botbot.me more than a year ago but no reply, but the code is open source so one of us could run it
04:19 < belcher> if anyone has hosting they're willing to volunteer, the bot can be found here https://github.com/botbotme
04:19 < belcher> i might post it on reddit/other places asking too
04:19 < btcdrak> try their IRC channel
04:20 < btcdrak> #lincolnloop
04:22 < belcher> trying now
04:24 <@waxwing> belcher: as i say, i *was* going to talk to you about it, but now i've more or less forgotten the details. iirc index_cache is overwitten only by update_cache_index, which happens at one point for tumbler flow (as you'd expect), but if you're not using 5 depths (which is not the default iirc), and you previously did use 5 depths (e.g. run yg), then the original 5th depth is lost.
04:24 <@waxwing> i *think* it was along those lines.
04:25 < belcher> its not good if the old one was overwritten
04:25 <@waxwing> i guess more common (much more) is: run tumbler second time after first time with shorter depth
04:26 <@waxwing> quite so, it should be "overlay" or whatever. but it's there in update_cache_index in Wallet
04:26 <@waxwing> i remember musing about it briefly but was more focused on just stopping the crash bug we had in 0.2.1 (or wherever it started)
04:26 <@waxwing> the bugfix i'm talking about is referenced in the 0.2.2 release notes
04:27 <@waxwing> huh, now i've written it out like that, really should have done it, if i described it right, it's a bad bug privacy-wise (at least for some small subset of use-cases). sorry just too much stuff.
04:28 < belcher> dont worry, write as much as you like :)
04:28 <@waxwing> i'm just sitting here thinking "damn, that is so simple, why didn't i just fix that!?"
04:29 <@waxwing> it's like 2 lines right, max
04:29 <@waxwing> https://github.com/JoinMarket-Org/joinmarket/blob/master/joinmarket/wallet.py#L234
04:30 < belcher> "How extremely stupid not to have thought of that myself!" ~Huxley
04:30 < belcher> also in joinmarket theres about a million other things to do so it doesnt always occupy our thoughts
04:30 <@waxwing> you get what i'm saying, though?
04:30 < belcher> nope
04:30 <@waxwing> see that ^ line. if index has len 4 and index_cache has len 5
04:31 <@waxwing> instead of overwrite, only overwrite first 4
04:31 < belcher> ah     
04:33 < belcher> ok got it
04:33 < belcher> in the case of multiple tumbler runs
04:33 < belcher> make it so it never loses information
04:33 <@waxwing> yes, technically also if yg. or any script uses index_cache.
04:35 <@waxwing> fortunately people almost always extend the list (go up the depths), not shorten, in practice. not that that makes it forgivable, but still :)
04:41 <@waxwing> to the extent that it's a bug, it's been there since the start, but i think it wasn't really a "bug" in as much as sync would pick up used addresses on startup. it's quite tangly to think about :)
04:42 <@waxwing> i think at least you could say, well, if we preserve information recorded in the cache, it will never be a bad thing.
06:36 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has joined #joinmarket     
06:58 -!- owowo [ovovo@gateway/vpn/mullvad/x-furilvcnmnzeullu] has quit [Ping timeout: 244 seconds]     
07:02 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has quit [Quit: Leaving.]     
07:14 -!- owowo [ovovo@gateway/vpn/mullvad/x-fyadileeykzqiimg] has joined #joinmarket     
07:14 -!- owowo [ovovo@gateway/vpn/mullvad/x-fyadileeykzqiimg] has quit [Changing host]     
07:14 -!- owowo [ovovo@unaffiliated/ovovo] has joined #joinmarket     
07:14 -!- owowo [ovovo@unaffiliated/ovovo] has quit [Changing host]     
07:14 -!- owowo [ovovo@gateway/vpn/mullvad/x-fyadileeykzqiimg] has joined #joinmarket     
07:19 -!- owowo [ovovo@gateway/vpn/mullvad/x-fyadileeykzqiimg] has quit [Ping timeout: 244 seconds]     
07:24 -!- owowo [~ovovo@unaffiliated/ovovo] has joined #joinmarket     
10:01 -!- coins123 [~coins123@unaffiliated/coins123] has quit [Ping timeout: 260 seconds]     
10:40 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 260 seconds]     
10:51 -!- belcher [~belcher@unaffiliated/belcher] has joined #joinmarket     
11:55 -!- coins123 [~coins123@unaffiliated/coins123] has joined #joinmarket     
12:48 < GithubBot5678> [joinmarket] anduck opened pull request #655: apt-get python-pip -> apt-get install python-pip (master...patch-2) https://git.io/vX6ei
15:39 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 260 seconds]     
15:52 -!- belcher [~belcher@unaffiliated/belcher] has joined #joinmarket     
15:58 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-uaqfbtlvvblcjumo] has quit [Quit: Connection closed for inactivity]     
17:05 -!- megaddin [aladdin@gateway/shell/fnordserver.eu/x-vjddsukbwybcefuo] has joined #joinmarket     
21:03 -!- goregrind [~goregrind@unaffiliated/goregrind] has quit [Ping timeout: 250 seconds]     
21:59 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-avdaurmuneuquceh] has joined #joinmarket