--- Day changed Mon Aug 05 2019
00:27 -!- Guadalupe28Okune [~Guadalupe@ns334669.ip-5-196-64.eu] has joined #joinmarket     
03:37 < waxwing> belcher_, would you be prepared to spend a little time reviewing #367? i looked it over and could do more, my only concern is not really the PR itself but rather than the fee checks have ended up too disjointed. there are several distinct checks in different places.
03:38 < waxwing> i want to at least try to finish #359 now i have a bit of time to devote to JM, it's fairly important but rather difficult.
06:38 -!- nixbitcoin [~nixbitcoi@gateway/tor-sasl/nixbitcoin] has joined #joinmarket     
06:43 < nixbitcoin> hey waxwing. I'm packaging joinmarket for nix-bitcoin and wondering how the scripts interact with the four joinmarket packages. Could I just copy the scripts folder over to the node and expect them to interact properly with jmbase, jmclient, jmbitcoin, and jmdaemon? Is there any way to use the scripts without virtualenv which adds an unnecessary layer to NixOS?
06:46 < waxwing> without venv? hmm. i guess you can just install the four JM packages to the system python
06:46 < waxwing> has anyone done that btw?
06:47 < waxwing> i mean something along those lines should work fine in principle, it'd just (from a usual perspective) be more of a pain to manage. maybe in your case it makes more sense, i dunno.
06:47 < waxwing> and then yeah as you say put the scripts/ directory "somewhere" and then operate as before.
06:48 < waxwing> oh and ofc you need all the dependencies that those pacakges depend on, in system python too. like twisted etc etc
06:48 < waxwing> i mean the installation script makes this literally one step, but it didn't occur to me that anyone would think virtualenv to be a bad thing.
06:49 < waxwing> that's basically how everyone uses python afaik, but that's maybe just a lack of knowledge on my part.
06:53 < waxwing> well for the dependencies i guess `pip install` has you covered anyway
07:03 -!- undeath [~undeath@hashcat/team/undeath] has joined #joinmarket     
07:40 -!- lnostdal [~lnostdal@77.70.119.51] has quit [Ping timeout: 245 seconds]     
08:28 < belcher_> waxwing: yes i will try to review it in the next few days
08:45 -!- Sentineo [~Undefined@unaffiliated/sentineo] has quit [Ping timeout: 245 seconds]     
08:54 -!- Sentineo [~Undefined@unaffiliated/sentineo] has joined #joinmarket     
09:45 < nixbitcoin> waxwing: it makes sense in our case because of the way NixOS manages packages.
09:45 < nixbitcoin> waxwing: The installation script is great but doesn't work for us because we are trying to use nixops to deploy the nix-bitcoin system which may or may not include joinmarket according to the users preference.
09:46 < nixbitcoin> waxwing: virtualenv isn't a bad thing in general, just AFAIK not needed for our use-case.
09:46 < nixbitcoin> waxwing: everyone except us :D
10:25 -!- deafboy [quasselcor@cicolina.org] has quit [Quit: deafboy]     
10:43 -!- MaxSan [~four@45.162.229.147] has joined #joinmarket     
11:56 -!- MaxSan [~four@45.162.229.147] has quit [Quit: Leaving.]     
12:05 < belcher_> after some more discussion on the ML (which isnt yet published due) iv become very concerned at the concept of renting out TXOs to other people for use in creating fidelity bonds
12:06 < belcher_> so iv been trying to come up with ways to make renting TXOs impossible, my latest email has two suggestions but neither are really that great
12:20 < waxwing> yeah the rent problem, i think i mentioned that somewhere because i saw someone say it on twitter. i wans't entirely sure how serious a problem it should be seen as.
12:26 < waxwing> undeath, ping
12:26 < waxwing> sorry ignore that ping on github, you weren't here earlier
12:27 < undeath> sup?
12:28 < waxwing> anyway i'll just ask. bci._get_used_indices gets indices of addresses in wallet, takes max of (current recorded index or highest used index). but would highest used index ever be higher?
12:28 < waxwing> the is_known_addr check is only going to check for things that are less than current index recorded in wallet (file)
12:28 < waxwing> because it checks _script_map which comes from populate_script_map which just fills out everything up to the recorded index I think?
12:33 < undeath> is_known_address is populated with scripts up to max_index + gap_limit
12:34 < undeath> which happens in _collect_addresses_init
12:34 < waxwing> undeath, are you sure about gap limit? populate script map only goes up to max index I think?
12:35 < undeath> https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/jmclient/jmclient/blockchaininterface.py#L597
12:36 < waxwing> yes those addresses are returned there, but i'm talking about the is_known_addr check which checks the _script_map
12:36 < waxwing> https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/jmclient/jmclient/blockchaininterface.py#L566
12:36 < undeath> wallet.get_new_addr will populate _script_map
12:37 < waxwing> ah yes, i see, thanks
12:38 < undeath> that's actually the main purpose of _collect_addresses_init
12:39 < belcher_> renting is a moderately serious problem, because it leads to far fewer makers
12:40 < belcher_> and so each maker has more privacy-relevant information that we hope they'll hide
12:42 < waxwing> undeath, so after the call to rewind wallet indices, if it was discovered that one of the addresses beyond the current max index was, in fact, used, then the call to _collect_addresses_gap will get addresses that weren't currently imported, right?
12:42 < waxwing> referring to https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/jmclient/jmclient/blockchaininterface.py#L537-L539
12:43 < waxwing> so you'd end up executing line 542 and wouldn't get to the gaplimit used check just below it
12:46 < waxwing> so .. why do we need to call add_watchonly_addresses (which quits) there on that line, wouldn't just import_addresses be enough?
12:46 < waxwing> because we already accounted for the addresses taht have been used.
12:47 < undeath> _check_gap_indices uses used_indices calculated by _get_used_indices before the _rewind_wallet_indices
12:48 < undeath> so the rewind shouldn't cause any screw-up with the gap limit
12:51 < waxwing> ok i didn't follow that part. check_gap_indices is only returning a boolean for gap_limit_used. i'm saying that rewind_wallet_indices followed by collect_addresses_gap means that the latter is starting with bumped max indices, if there were addresses used, according to used_indices, in the gap.
12:51 < waxwing> and then 542 would be triggered (I think)
12:52 < undeath> the idea behind the if/elif/else in lines 540+ is: if it was detected that the gap limit had been used our internally saved indices are outdated and since we just used the gap limit it may be necessary to scan the full gap limit once more in an additional step
12:53 < waxwing> yes but i'm talking about a case where i think that check never happens, because new_addresses is not a subset of imported at 540-542
12:54 < undeath> uh, i think there's a bug in _check_gap_indices, looking at the code
12:54 < waxwing> this all started by kristapsk talking about a case where he had funded an external and he got the quit and maybe rescan message, and i didn't think he should
12:54 < undeath> it will return False basically after the first check, almost never actually hitting the True in any case
12:55 < undeath> if there is an early abort in that method it should be a "return True", not a "return False"
12:56 < waxwing> i don't get it. it will return False if any of the used are above the max, and return True if none of them are.
12:57 < undeath> I might be confusing things rn
12:57 < undeath> let me check again
12:58 < undeath> oh, nvm
12:58 < undeath> i missed that the result of that call is negated
12:59 < waxwing> yeah
13:00 < waxwing> so what i'm saying this (and i know this is stupid hard to communicate about): rewind_wallet_indices can potentially bump the wallet's index, if the used-indices are above the existing indices.
13:00 < waxwing> then after that, we call collect addresses gap, which will collect "gap" amount beyond the currently set index (which was just bumped)
13:00 < waxwing> and then new_addresses can be (will be) not a subset of imported, because we went further than the initial gap.
13:00 < waxwing> and so i think it'd hit 542.
13:01 < waxwing> and i don't see why that needs to quit, i thihnk it can just import_addresses instead, because we already accounted for used addresses, so no need for rescan.
13:01 < waxwing> something like that.
13:01 -!- nixbitcoin [~nixbitcoi@gateway/tor-sasl/nixbitcoin] has quit [Quit: nixbitcoin]     
13:11 < undeath> i suspect my train of thought went like this while writing that code: after we have used the gap limit we will have to know the status of the new gap limit (after considering the recent change), hence we need to import addresses up to that new gap limit and rescan in order to not miss any used addresses
13:14 < waxwing> but we checked the used addresses, right.
13:14 < waxwing> line 530
13:15 < waxwing> or ... hmm, because we only checked up to the gap limit
13:15 < undeath> exactly
13:15 < waxwing> because any further and then the wallet will not recognize it as being owned (is_known_addr)
13:16 < waxwing> yes this relates to a vague thought i had earlier. that if we cannot assume that the wallet label uniquely contains transactions for this Joinmarket wallet, it makes the logic more difficult.
13:16 < undeath> i can definitely see the problem here now that you explained it
13:17 < waxwing> yes something needs to change, but this is tricky as hell.
13:17 < undeath> i'm just not sure right now how to ideally handle that
13:17 < waxwing> i am trying to rewrite this code, at least partly. i started in #359 with something simple that handles a lot of erroneous restarts. but it doesn't handle everything.
13:18 < waxwing> i am lothe to completely rewrite, jsut for the hell of it, but i also am not sure exactly what kind of tweaking would resolve any possible problem. hard.
13:18 < waxwing> i actually primarily wanted to create a live update style monitoring so that things like deposits-while-running would be handled seamlessly.
13:18 < waxwing> but i really want to get rid of exceptional errors like the one we discuss here.
13:20 < undeath> yes, I also notice some problems there that were a little annoying
13:20 < undeath> because this for example also triggers when you restart a yg after a successful cj
13:25 < waxwing> yeah but that was addressed by just "importing ahead to the gap" for every import (see #359 commit), but that was just a "patch up" kind of change.
13:26 < undeath> ah     
13:26 < waxwing> wait, would that exact condition trigger for a successful coinjoin? you update the index then.
13:27 < undeath> actually I think it would because the sync code would detect that not all addresses until the gap limit have been imported
13:27 < waxwing> i think the exact one we're talking about happens only if you use an address in the gap as it currently stands.
13:27 < undeath> (which is ofc not the intended behaviour)
13:29 < undeath> yes, i think that's a different issue from the one kristapsk observed
13:31 < undeath> it kinda makes sense for the case where you fund an external address to see the behaviour, because jm cannot discern between a wrongly set index and a newly funded address
13:32 < undeath> i think that's what the problem boils down to
13:41 < waxwing> perhaps we could check the list of addresses which are used, but which are not found to be in the wallet up to the gap limit, and flag that as an error condition, because that shouldn't happen.
13:41 < waxwing> people shouldn't fund beyond the gap limit, because that isn't even displayed.
13:41 < undeath> it also happens if you fund within the gap limit
13:42 < undeath> jm thinks your index is i while after funding your index is actually i+1, which can only be detected by scaning the gap limit, thus triggering the restart msg
13:43 -!- Sentineo [~Undefined@unaffiliated/sentineo] has quit [Quit: leaving]     
13:47 < waxwing> undeath, yeah but i'm trying to argue that what i was proposing above ought to address that if we could discount the possibility that there were any used addresses beyond the gap limit.
13:47 < waxwing> concretely, that line 542 could be import_addresses and not add_watch_only_addresses (quitting) if we knew that there was no further usage beyond the gap limit.
13:48 < undeath> oh yeah, that would get rid of the problem
13:48 < undeath> there might be some (weird, unusual) cases where the sync might not detect all used addresses
13:48 < undeath> but that's probably acceptable
13:49 < undeath> if you recover a wallet you should just set a very high gap limit and in other cases that should not happen at all
13:51 < undeath> mh, what about a case where a long-running yg crashes and doesn't update wallet indices after using lets say 100 addresses?
13:51 < waxwing> also like i mentioned, if there were further used addresses detected on line 530 that didn't get caught, it could flag an error.
13:51 < waxwing> undeath, i'd have to check but i might have added a persist step for individual coinjoin transactions.
13:51 < waxwing> certainly we should keep it up to date on a per tx basis, no.
13:52 < undeath> certainly can't hurt
13:58 -!- arubi [~ese168@93-173-92-91.bb.netvision.net.il] has joined #joinmarket     
13:58 -!- arubi [~ese168@93-173-92-91.bb.netvision.net.il] has quit [Remote host closed the connection]     
14:00 -!- arubi [~ese168@unaffiliated/ese168] has joined #joinmarket     
14:05 -!- arubi [~ese168@unaffiliated/ese168] has quit [Quit: Leaving]     
14:06 -!- lnostdal [~lnostdal@77.70.119.51] has joined #joinmarket     
14:06 -!- arubi_ [~ese168@gateway/tor-sasl/ese168] has joined #joinmarket     
14:06 -!- arubi_ is now known as arubi     
14:09 < arubi> oof.  my cert expired.  guess I'm back to my usual lurking :)  hope everybody are well
14:57 -!- undeath [~undeath@hashcat/team/undeath] has quit [Quit: WeeChat 2.4]     
15:01 < harding> belcher_: regarding your most recent bitcoin-dev post, if fidelity bonds can be secured by multisig (which seems like a option to me), then Mallory and Malfoy can create a single UTXO that neither of them controls using the Spillman method (create multisig address, create unsigned spend to that address, create and sign (but do not broadcast) spend from that address back to their individual addresses, 
15:01 < harding> then sign and broadcast the deposit to the multisig address).
15:02 < harding> seems like a *useful* option to me*
15:32 -!- Guadalupe28Okune [~Guadalupe@ns334669.ip-5-196-64.eu] has quit [Ping timeout: 245 seconds]     
15:34 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]     
15:35 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
15:41 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]     
15:42 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
16:12 < waxwing> arubi, hi :)
16:53 -!- MaxSan [~four@45.162.229.147] has joined #joinmarket     
16:56 -!- azizLIGHT [~azizLIGHT@unaffiliated/azizlight] has quit [Ping timeout: 258 seconds]     
17:00 -!- Cory [Cory@unaffiliated/cory] has joined #joinmarket     
17:16 -!- azizLIGHT [~azizLIGHT@unaffiliated/azizlight] has joined #joinmarket     
17:36 -!- AgoraRelay [~jmrelayfn@p5DE4AD7B.dip0.t-ipconnect.de] has quit [Ping timeout: 258 seconds]     
17:36 -!- CgRelayBot [~CgRelayBo@p5DE4AD7B.dip0.t-ipconnect.de] has quit [Ping timeout: 245 seconds]     
17:45 -!- CgRelayBot [~CgRelayBo@p5DE4AC41.dip0.t-ipconnect.de] has joined #joinmarket     
17:46 -!- AgoraRelay [~jmrelayfn@p5DE4AC41.dip0.t-ipconnect.de] has joined #joinmarket     
17:58 -!- MaxSan [~four@45.162.229.147] has quit [Ping timeout: 268 seconds]     
18:21 -!- MaxSan [~four@45.162.228.171] has joined #joinmarket