--- Day changed Sat Nov 12 2016 00:02 < GithubBot5678> [joinmarket] WyseNynja opened pull request #654: Fix install deps (develop...patch-1) https://git.io/vXKuP 00:27 -!- windsok [~windsok@45.63.59.8] has quit [Ping timeout: 244 seconds] 00:32 -!- windsok [~windsok@45.63.59.8] has joined #joinmarket 01:02 -!- windsok [~windsok@45.63.59.8] has quit [Ping timeout: 260 seconds] 01:26 -!- coins123 [~coins123@unaffiliated/coins123] has joined #joinmarket 01:59 -!- windsok [~windsok@45.63.59.8] has joined #joinmarket 03:44 < belcher> --fast might need to be renamed, since its a verb with no noun 03:44 < belcher> tumbler.py --fast looks like it might make your tumbling faster or something 03:44 < belcher> --fast-sync instead? 03:44 < btcdrak> yes, that's a point.. but you should really think about making --fast-syn the default 03:45 < btcdrak> and add another switch for --deep/legacy/slow-sync or whatever you want to call it. 03:47 < belcher> --fast-sync only works if you're already sync'd 03:48 < btcdrak> if you arent synced and you use that, does it spew an error? 03:48 < belcher> hmm 03:48 < belcher> i didnt write it but i think it might not be possible to tell 03:48 < belcher> but if it is, that would be good 03:48 < belcher> sec, looking at the code 03:49 < btcdrak> if you can detect it just do an original sync (spit out a message, "performing initial sync, will take a while..." 03:56 < belcher> looking at the code, it could be detected i think if you check whether wallet.index_cache is not all zeros, and/or whether wallet_name is an existing label in bitcoin core, but ill let waxwing confirm since he wrote the code 03:56 < belcher> the commit that added it is here btw https://github.com/JoinMarket-Org/joinmarket/commit/0281900e5bfbd2720ccae6b7546e6bad303d28da 04:01 <@waxwing> belcher: yes. basically, as i mentioned in the release notes, if it's not already synced such that there isn't already an index_cache, then --fast can't work. 04:02 <@waxwing> if there *is* an index cache, great, but fast is assuming it's up to date. 04:02 <@waxwing> if it's not, you'll get an incorrect balance. 04:02 <@waxwing> the whole thing is very tricky not least because of the various scenarios: recovering a wallet, a new Core instance, an index_cache which is not accurate, whether the gap limit is or is not sufficient. 04:03 <@waxwing> and as we mentioned on the issue at the time (so saying for btcdrak 's benefit), the reason i applied --fast as an explicit option is so that an unsuspecting user doesn't have it being used, reporting incorrect balance, and not knowing why. 04:03 < belcher> hmm yes, so a wrong index_cache might happen if you have the same wallet on two separate instances 04:04 <@waxwing> the history of this is pretty complicated. belcher 's original code worked fine by not assuming anything, and building up an accurate sync from a series of steps. but then we started to get more and more spam requests from spy/spies. 04:05 <@waxwing> started to get complaints of "im having to run sync over and over" and we started hacking around addr_req_count. 04:05 <@waxwing> so i "fixed" that by using index_cache where possible to deduce how many imports were needed, but on reflection it seems it wasn't a good solution (albeit it did solve the endless re-run sync problem) 04:06 < belcher> as well as spies, it was also just slow 04:06 <@waxwing> then someone else pointed out you can query history with getaddressusages, and so --fast allows a person with a big wallet to do the whole thing cleanly, *assuming* index_cache is up to date. 04:07 <@waxwing> i feel like it would be good to rewrite the whole thing from scratch .. it should be possible for it to be cleaner, *although* don't assume it'll necessarily be *really* simple, because this kind of two-layer 'watch only address and then JM HD wallet on top' thing, plus the fact that we inevitably use tons of addresses, makes it innately at least somewhat complex. 04:09 <@waxwing> (minor point but i often wondered if index/index_cache maybe was not necessary to have as two variables; i.e. you could just treat the data in file as the persistence/cache of index) 04:09 <@waxwing> but even if so that doesn't matter really, that's not what makes it complicated. 04:10 < belcher> right so at the start the code just does index = index_cache 04:11 <@waxwing> if that's the case, it's more just about readability i guess 04:11 < belcher> i cant really think of any situations where index_cache would be wrong, except where someone is running the same wallet twice on different instances 04:12 <@waxwing> yeah i guess so, it's really about people switching Core instances, or also i guess they could make mistakes (like use an earlier backup)? 04:12 <@waxwing> well, maybe that's not even a mistake, if you lost the most up to date wallet.json, it's not a mistake to use an earlier one i think :) 04:12 < belcher> ok 04:13 <@waxwing> there was also a confusion about what happens with tumbler, because index_cache length gets changed by it. 04:13 < belcher> wallet.json isnt really a file you back up, your backup is the seed 04:13 <@waxwing> actually i was going to talk to you about that 04:13 <@waxwing> yes i know but people do that kind of thing 04:13 < belcher> yes 04:18 < belcher> another thing is id like an irc logging bot here, so the stuff we talk about is recorded and able to be referenced 04:18 < belcher> i put in a request to botbot.me more than a year ago but no reply, but the code is open source so one of us could run it 04:19 < belcher> if anyone has hosting they're willing to volunteer, the bot can be found here https://github.com/botbotme 04:19 < belcher> i might post it on reddit/other places asking too 04:19 < btcdrak> try their IRC channel 04:20 < btcdrak> #lincolnloop 04:22 < belcher> trying now 04:24 <@waxwing> belcher: as i say, i *was* going to talk to you about it, but now i've more or less forgotten the details. iirc index_cache is overwitten only by update_cache_index, which happens at one point for tumbler flow (as you'd expect), but if you're not using 5 depths (which is not the default iirc), and you previously did use 5 depths (e.g. run yg), then the original 5th depth is lost. 04:24 <@waxwing> i *think* it was along those lines. 04:25 < belcher> its not good if the old one was overwritten 04:25 <@waxwing> i guess more common (much more) is: run tumbler second time after first time with shorter depth 04:26 <@waxwing> quite so, it should be "overlay" or whatever. but it's there in update_cache_index in Wallet 04:26 <@waxwing> i remember musing about it briefly but was more focused on just stopping the crash bug we had in 0.2.1 (or wherever it started) 04:26 <@waxwing> the bugfix i'm talking about is referenced in the 0.2.2 release notes 04:27 <@waxwing> huh, now i've written it out like that, really should have done it, if i described it right, it's a bad bug privacy-wise (at least for some small subset of use-cases). sorry just too much stuff. 04:28 < belcher> dont worry, write as much as you like :) 04:28 <@waxwing> i'm just sitting here thinking "damn, that is so simple, why didn't i just fix that!?" 04:29 <@waxwing> it's like 2 lines right, max 04:29 <@waxwing> https://github.com/JoinMarket-Org/joinmarket/blob/master/joinmarket/wallet.py#L234 04:30 < belcher> "How extremely stupid not to have thought of that myself!" ~Huxley 04:30 < belcher> also in joinmarket theres about a million other things to do so it doesnt always occupy our thoughts 04:30 <@waxwing> you get what i'm saying, though? 04:30 < belcher> nope 04:30 <@waxwing> see that ^ line. if index has len 4 and index_cache has len 5 04:31 <@waxwing> instead of overwrite, only overwrite first 4 04:31 < belcher> ah 04:33 < belcher> ok got it 04:33 < belcher> in the case of multiple tumbler runs 04:33 < belcher> make it so it never loses information 04:33 <@waxwing> yes, technically also if yg. or any script uses index_cache. 04:35 <@waxwing> fortunately people almost always extend the list (go up the depths), not shorten, in practice. not that that makes it forgivable, but still :) 04:41 <@waxwing> to the extent that it's a bug, it's been there since the start, but i think it wasn't really a "bug" in as much as sync would pick up used addresses on startup. it's quite tangly to think about :) 04:42 <@waxwing> i think at least you could say, well, if we preserve information recorded in the cache, it will never be a bad thing. 06:36 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has joined #joinmarket 06:58 -!- owowo [ovovo@gateway/vpn/mullvad/x-furilvcnmnzeullu] has quit [Ping timeout: 244 seconds] 07:02 -!- Giszmo [~leo@pc-40-227-45-190.cm.vtr.net] has quit [Quit: Leaving.] 07:14 -!- owowo [ovovo@gateway/vpn/mullvad/x-fyadileeykzqiimg] has joined #joinmarket 07:14 -!- owowo [ovovo@gateway/vpn/mullvad/x-fyadileeykzqiimg] has quit [Changing host] 07:14 -!- owowo [ovovo@unaffiliated/ovovo] has joined #joinmarket 07:14 -!- owowo [ovovo@unaffiliated/ovovo] has quit [Changing host] 07:14 -!- owowo [ovovo@gateway/vpn/mullvad/x-fyadileeykzqiimg] has joined #joinmarket 07:19 -!- owowo [ovovo@gateway/vpn/mullvad/x-fyadileeykzqiimg] has quit [Ping timeout: 244 seconds] 07:24 -!- owowo [~ovovo@unaffiliated/ovovo] has joined #joinmarket 10:01 -!- coins123 [~coins123@unaffiliated/coins123] has quit [Ping timeout: 260 seconds] 10:40 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 260 seconds] 10:51 -!- belcher [~belcher@unaffiliated/belcher] has joined #joinmarket 11:55 -!- coins123 [~coins123@unaffiliated/coins123] has joined #joinmarket 12:48 < GithubBot5678> [joinmarket] anduck opened pull request #655: apt-get python-pip -> apt-get install python-pip (master...patch-2) https://git.io/vX6ei 15:39 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 260 seconds] 15:52 -!- belcher [~belcher@unaffiliated/belcher] has joined #joinmarket 15:58 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-uaqfbtlvvblcjumo] has quit [Quit: Connection closed for inactivity] 17:05 -!- megaddin [aladdin@gateway/shell/fnordserver.eu/x-vjddsukbwybcefuo] has joined #joinmarket 21:03 -!- goregrind [~goregrind@unaffiliated/goregrind] has quit [Ping timeout: 250 seconds] 21:59 -!- btcdrak [uid165369@gateway/web/irccloud.com/x-avdaurmuneuquceh] has joined #joinmarket