--- Day changed Wed Jun 29 2016
00:12 -!- coins123 [~coins123@unaffiliated/coins123] has joined #joinmarket
00:17 -!- molz [~molly@unaffiliated/molly] has joined #joinmarket
00:17 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 246 seconds]
00:21 -!- fqtw__ is now known as fqtw
00:24 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has quit [Remote host closed the connection]
00:49 -!- p15 [~p15@150.91.145.64.unassigned.bringover.net] has joined #joinmarket
01:21 < adlai> here's an idea for frustrated yield generators!
01:21 < adlai> cache the last-seen balance in a wallet
01:21 < adlai> optionally, keep scanning until you hit a greater balance
01:22 < adlai> ... more precise test: look for one of the cjout addresses in the transactions draining the wallet
01:22 < adlai> (if you are missing the last addresses in bitcoind, you'll still have some final transactions in listtransactions)
01:42 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has quit [Ping timeout: 272 seconds]
02:26 -!- berndj [~berndj@197.242.93.82] has quit [Ping timeout: 246 seconds]
02:40 -!- mkarrer [~mkarrer@48.red-83-47-111.dynamicip.rima-tde.net] has quit []
02:42 -!- mkarrer [~mkarrer@48.red-83-47-111.dynamicip.rima-tde.net] has joined #joinmarket
02:46 -!- p15 [~p15@150.91.145.64.unassigned.bringover.net] has quit [Ping timeout: 250 seconds]
04:02 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has joined #joinmarket
04:32 -!- berndj [~berndj@azna.co.za] has joined #joinmarket
04:54 -!- lnostdal [~lnostdal@64-81-11.connect.netcom.no] has joined #joinmarket
07:26 -!- Giszmo [~leo@pc-122-14-46-190.cm.vtr.net] has joined #joinmarket
08:44 -!- core [~core@unaffiliated/core] has quit [Ping timeout: 244 seconds]
08:45 -!- core [~core@unaffiliated/core] has joined #joinmarket
08:52 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 252 seconds]
08:54 < raedah> adlai: dont even need to cache the last-seen balance. you can pull the exact expected balance from bitcoin-core.
08:55 < raedah> but even all that sounds like just an extra failsafe. The wallet index already gives us a cache of where we are suppose to be. We just have code that doesnt use the wallet index.
08:56 < raedah> May 12th, https://github.com/JoinMarket-Org/joinmarket/issues/522#issuecomment-218871865
08:57 < raedah> "The code that causes this is these 15 lines here: https://github.com/JoinMarketOrg/joinmarket/blob/master/joinmarket/blockchaininterface.py#L591-L607
08:57 < raedah> "It just imports 20 addresses for internal and external and for each mixdepth level, so 200 total. Instead of doing that, it should be using the wallet index_cache numbers. It can be the wallet index_cache number + 20 for each level that it imports.
09:08 -!- moli [~molly@unaffiliated/molly] has joined #joinmarket
10:44 < waxwing> raedah: in case of not recovery, i.e. you have wallet.json, then i *think* i agree that we can just import all up to index_cache at the start without batching. i'd like to hear belcher_ 's view.
10:44 < waxwing> but, just for clarification of discussion, of course index_cache *is* used in the following code (is_index_ahead_of_cache)
10:45 < waxwing> in case of recovery, i.e. no index_cache available, then the sort of multi-stage approach in use now is i guess unavoidable; you make a sensible look-forward guess, import, check for used addresses and do the gap limit check. if necessary, go round and import again.
10:47 < waxwing> but i might have missed something.
10:47 < waxwing> i'll link this again in case it helps anyone thinking about it: http://0bin.net/paste/gCSoYSCCEcOsCYmo#3ZjlEicKTeFAK+y9Br1ZH40GWgmMqgR2YAHe1wNbVef
10:54 < raedah> waxwing: is_index_ahead_of_cache is not called in the lines I pointed to, https://github.com/JoinMarketOrg/joinmarket/blob/master/joinmarket/blockchaininterface.py#L591-L607
10:54 < waxwing> that's why i said "in the following code"
10:55 < raedah> get_new_addr is called, shortly after that the program quits.   is_index_ahead_of_cache is never hit.
10:55 < waxwing> yes, if it fails; i do understand :)
10:56 < waxwing> i was emphasising for clarification of those trying to follow the discussion that syncing does not just ignore the cache. or, trying to :)
10:56 -!- OverlordQ [~ovrlrdq@mediawiki/OverlordQ] has quit [Remote host closed the connection]
10:57 < raedah> and I am emphasising for clarification, that is_index_ahead_of_cache is not being hit, and is therefor the cache is being ignored.
10:57 < waxwing> yes but that's just how the algo works now. 
10:57 < waxwing> it tries importing in batches and sees if the wallet addr list is included in the imported address list.
10:58 < raedah> as oppose to how it will work in the future when the bug is fixed?
10:58 < waxwing> step 3 in the list of 13 steps in the 0bin.
10:58 < waxwing> what specific bug do you mean?
10:58 < raedah> so lets get to the bottoms of this.
10:58 < raedah> is the cache being ignored, or is it not being ignored?
10:58 < waxwing> when i said "as opposed" i meant as opposed to your suggested approach, which I *think* makes sense.
11:03 < raedah> the code requests 20 addresses per run. It should be using the cache which for example would say 500 addresses, but it is ignoring it. The user would have to start their yield generator 25 times, and would likely have to do a full blockchain rescan each of the 25 times as well. That would take all day. Or we could just import the known 500 addresses right off the bat and save the user a day of repetetive
11:03 < raedah> clicking, assuming they actually know they are suppose to do that.
11:04 < raedah> hence why I said in the github comment, "change line #591 addr_req_count from the default of 20 to something higher like 500, where 500 is the highest index in your wallet file and then you will only have to do one round of imports and one bitcoind rescan."
11:05 < waxwing> it's ultra confusing. notice that this: https://github.com/JoinMarket-Org/joinmarket/blob/master/joinmarket/blockchaininterface.py#L594-L596 is always going to get the *first* batch of addresses on each run, because the Wallet constructor sets the wallet.index to all zeros.
11:05 < waxwing> while the imported address list returned by 'getaddressesbyaccount' is going to return an increasing list each time, since you actually did imports into the wallet.
11:06 < waxwing> raedah: i was never trying to argue against increasing that fixed number. i was really mostly concerned with making sure an infinite loop is not possible.
11:06 < waxwing> but more specifically re: what you said today, like i said, i *think* i agree that importing all up to index_cache value immediately should make sense. 
11:07 < waxwing> in the non-recovery case
11:07 < waxwing> full blockchain rescan each of 25 times: no, why do you say that?
11:11 < waxwing> correct me if i'm wrong, but i *think* you might have the issue of needing to rescan multiple times if you changed to a new bitcoind (so new wallet.dat). but you don't need it if you haven't changed the bitcoind instance you're using.
11:11 < raedah> even without needing a full rescan each time, running the script lots of time to get the addresses importing is not correct. it should import the amount from the cache.
11:12 < waxwing> if i got that right, then perhaps that illustrates why what you're describing is ultra important - in the case where you've changed your bitcoind instance, it could be a big deal.
11:12 < raedah> yes, I am talking about with a new bitcoind.
11:12 < waxwing> raedah: well i'm just repeating myself now, but as you can see from my first comments today, i *think* i agree, but i'd like to hear belcher_ 's comment.
11:12 < raedah> if you have the same bitcoin (wallet.dat) then there is no issue at all, because all your addresses are already imported, and already in sync with your cache. it just runs as normal.
11:13 < waxwing> well, you can get a situation where you need multiple runs of wallet-tool even with same bitcoind. happened to me quite often.
11:13 < waxwing> but, at this point, i think we're largely on the same page.
11:14 -!- King_Rex [~King_Rex@unaffiliated/king-rex/x-3258444] has joined #joinmarket
11:14 < waxwing> i was originally more concerned with whether there might be an infinite loop issue, but it seems not. there seems to be just this one main issue: that you really don't want to be batching if starting "from scratch".
11:14 < raedah> okay, also FYI, I emperically verified this and changing  addr_req_count to the number from the cache works. one script run with imports, one rescan, and its up and running.
11:14 < waxwing> this is another reason it's all so confusing: multiple situations: have wallet.json but new bitcoind, don't have wallet.json and don't have bitcoind, etc etc
11:15 < waxwing> like, imagine you're recovering on a new bitcoind: then you have no index_cache, so you don't know how much to import before doing listtransactions.
11:15 < waxwing> i suppose in that case, it's mostly a case of being ultra-generous with gap limits and import batching.
11:16 < raedah> its quite easy to backup the wallet.json. its a small file.  bitcoind on the other hand takes gigs to backup and should not even need to be backed up, because the addresses _should_ just reimport.
11:16 < waxwing> but i don't think there could be a perfectly clean solution, in that worst case.
11:16 < waxwing> yes good point. but still some people will lose the files.
11:16 < waxwing> after all, we provide a seed.
11:16 < raedah> so you want to make it work well even without the cache
11:17 < waxwing> well any improvement is good; so if that worst case can't be improved much, so be it. just trying to understand the full set of possibilities/problems.
11:20 < waxwing> the obvious concrete thing we could do is implement your idea of "use index_cache instead of addr_req_count if available". thus for the new bitcoind instance case but having wallet.json, you get the improvement. i think (and hope!) that's correct, so then if i understood right, you have exactly one rescan to do.
11:21 < waxwing> i suppose all this nastiness is in some sense a byproduct of having a 2-layer wallet.
11:21 < raedah> yes, we are on the same page now.
11:22 < raedah> related, https://github.com/JoinMarket-Org/joinmarket/issues/478
11:23 < waxwing> that code edit has to be done while bearing in mind the case of recovery without wallet.json. (maybe it's just a case of checking whether index_cache exists, haven't looked at it yet)
11:25 < waxwing> i don't understand #478.
11:27 < raedah> just a situation where the addresses are being scanned and and the new balances being displayed, but the cache number is not being updated in the json file.
11:28 < waxwing> ah, yes, think i see what you mean.
12:00 -!- mkarrer_ [~mkarrer@48.red-83-47-111.dynamicip.rima-tde.net] has joined #joinmarket
12:02 -!- mkarrer [~mkarrer@48.red-83-47-111.dynamicip.rima-tde.net] has quit [Ping timeout: 244 seconds]
12:09 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 250 seconds]
12:26 -!- King_Rex [~King_Rex@unaffiliated/king-rex/x-3258444] has quit [Remote host closed the connection]
12:27 -!- King_Rex [~King_Rex@unaffiliated/king-rex/x-3258444] has joined #joinmarket
12:27 -!- King_Rex [~King_Rex@unaffiliated/king-rex/x-3258444] has quit [Remote host closed the connection]
14:14 -!- arubi [~ese168@unaffiliated/arubi] has quit [Ping timeout: 276 seconds]
14:27 -!- arubi [~ese168@unaffiliated/arubi] has joined #joinmarket
14:37 -!- grubles [~grubles@unaffiliated/grubles] has quit [Quit: brb]
14:53 -!- grubles [~grubles@unaffiliated/grubles] has joined #joinmarket
14:53 -!- moli [~molly@unaffiliated/molly] has joined #joinmarket
15:29 -!- arubi [~ese168@unaffiliated/arubi] has quit [Ping timeout: 250 seconds]
15:33 -!- berndj [~berndj@azna.co.za] has quit [Quit: ZNC - http://znc.in]
15:42 -!- arubi [~ese168@unaffiliated/arubi] has joined #joinmarket
16:26 -!- berndj [~berndj@197.242.93.84] has joined #joinmarket
17:02 -!- lnostdal [~lnostdal@64-81-11.connect.netcom.no] has quit [Quit: lnostdal]
18:04 -!- molz [~molly@unaffiliated/molly] has joined #joinmarket
18:07 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
18:58 -!- fqtw_ [~me@x4d0b957c.dyn.telefonica.de] has joined #joinmarket
19:01 -!- fqtw [~me@x5d803633.dyn.telefonica.de] has quit [Ping timeout: 250 seconds]
19:03 -!- p15 [~p15@168.91.145.64.unassigned.bringover.net] has joined #joinmarket
19:28 -!- dingus [~grubles@unaffiliated/grubles] has joined #joinmarket
21:20 < raedah> just found this joinmarket shoutout http://www.youtube.com/watch?v=YVlQE-ObEXk&t=14m0s
21:31 -!- moli [~molly@unaffiliated/molly] has joined #joinmarket
21:33 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
21:37 -!- fqtw_ is now known as fqtw
22:12 -!- waxwing [~waxwing@62.205.214.125] has quit [Ping timeout: 260 seconds]
22:41 -!- waxwing [~waxwing@62.205.214.125] has joined #joinmarket
23:56 < fluffypony> raedah: yup, I ran the slide past waxwing before my talk :)