--- Day changed Tue Oct 22 2019
01:22 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has quit [Remote host closed the connection]     
01:39 -!- Zenton [~user@unaffiliated/vicenteh] has joined #joinmarket     
01:43 -!- Giszmo [~leo@122-58-98-6-adsl.sparkbb.co.nz] has joined #joinmarket     
01:43 -!- Giszmo [~leo@122-58-98-6-adsl.sparkbb.co.nz] has quit [Client Quit]     
03:25 -!- ircuser [b94187bc@gateway/web/cgi-irc/kiwiirc.com/ip.185.65.135.188] has joined #joinmarket     
03:29 -!- ircuser [b94187bc@gateway/web/cgi-irc/kiwiirc.com/ip.185.65.135.188] has quit [Remote host closed the connection]     
05:37 -!- AgoraRelay [~jmrelayfn@p5DE4AA06.dip0.t-ipconnect.de] has joined #joinmarket     
06:02 -!- willcl_ark [~quassel@cpc123762-trow7-2-0-cust7.18-1.cable.virginm.net] has joined #joinmarket     
06:21 < waxwing> arubi, just thought i'd mention, there's been some discussion of SNICKER on the mailing list (and a ton offline too in Berlin and with a couple other people too), I just added one more response into there, and thought I'd ping you as I think you'll be interested in the problem raised.
06:22 < waxwing> Problem being basically: incompatibility with watch-only.
06:23 -!- puddinpop [~puddinpop@unaffiliated/puddinpop] has quit [Ping timeout: 250 seconds]     
06:23 < harding> waxwing: if you posted to the mailing list, it hasn't arrived here yet.  (Just an FYI for arubi)
06:23 < waxwing> harding, ok cool. well it was like 2 minutes ago so i guess that's normal :)
06:23 < waxwing> but i'd forgotten that it takes a bit yeah, good point
06:25 < harding> Something I replied to ghost43 off-list was that LN transactions have basically the same problem wrt watch-only wallets as snickers.
06:26 < waxwing> yeah i mentioned that too :)
06:26 < harding> Ha.     
06:26 < waxwing> i mean the analogy won't be perfect but we're in that same ballpark
06:26 < waxwing> the other thing i mentioned was to distinguish between monitoring and recovering
06:27 < waxwing> because the tweaks are privacy controlling, but not money controlling, secrets
06:27 < waxwing> so a cold setup could still be given tweaks. maybe that was laready mentioned, not sure, wanted to clear it up.
06:29 < harding> Yeah, I think I mentioned that on-list, but you probably said it better.  Oh, your email just arrived.  /me reads
06:30  * waxwing frowns at harding for saying 'snickers'
06:34 < harding> waxwing: re the email, I don't know that it's fair to say that c is only a privacy-controlling secret.  If you don't know the tweak, you can't spend your money, right?
06:36 < harding> E.g., if the cold wallet doesn't have access to the tweaks because they were derived from random data and were lost when the watch-only wallet was accidentally deleted, that money is gone.
06:37 < waxwing> grr yeah that is a fair point, i mean (as i'm sure you got) .. it's not a secret that needs to be hidden to keep access to your money.
06:38 < waxwing> or to control access .. anyway. yeah i thought it encapsulated the point well but it doesn't. i'd hope people understood anyway.
06:40 < harding> Yeah.  Your point provides the insight that you could possibly get the secret from the counterparty if you lost the ability to regenerate it yourself.  Maybe there's some fancy smart contract way of paying them for it too, e.g. "I see one of my outputs went to this UTXO that looks like it was created a snicker coinjoin; if you can provide a tweak that combines with one of my pubkeys, you get 10% of the funds."
06:41 < harding> But I suspect we're already way past the line of things anyone will ever implement.  :-)
06:44 < waxwing> oh. well indeed that's not practical but it's a very interesting thought. like a market for tweaks or something heh.
06:48 < waxwing> it would be nice to shoehorn in SNICKER to LN channel opens, it's natural in a way (coinjoining channel opens is a nice idea that's been discussed), but i'm not sure how you could make it work given timing matters with LN. i guess basically not quite unless there was some massively active market for SNICKER joins. oh well.
09:10 < arubi> waxwing, ah cool will be following up on the thread (when I see it on the web interface), but I /did/ think about a setup that lets you back up your own tweaks, as long as you use deterministic generation of k for the signature.  basically, you compute 'k1 = rfc6979(d,z)' as usual, then do 'k2 = k1 + c' (c is the tweak), and sign using k2 as the nonce.  comes time to recover from backup, you can generate k1 again from knowing (d,z), and can 
09:10 < arubi> solve for k2 (since you know d), then just 'c = k2 - k1'
09:11 < arubi> kinda like sign-to-contract, but to yourself
09:14 < arubi> oh and harding too ^ :)
09:23 < arubi> oh but now I'm not sure if I understood the real issue correctly.  right I'll follow up on the thread as it comes in
09:24 -!- bsm1175321 [~mcelrath@pool-70-109-136-73.cncdnh.east.myfairpoint.net] has joined #joinmarket     
10:39 < waxwing> arubi, the issue with what you're saying, as i understand it, is that what's been raised is, how do you recover deterministically (or monitor) without private keys; to regenerate signature nonces requires private keys (so 'd' here).
11:20 < arubi> well we have to assume that to recover deterministically you at least have the master privkey right?  what else could we work with?
11:21 < arubi> monitor with only say a master xpub..  yea that's harder, but need to think about it
11:24 < arubi> but really I still think I'm missing something about the point raised (haven't read the emails yet), we already do generate tweaks by manner that is recoverable as long as you have the master xprv for the "native" wallet (just normal privkeys without tweaks)
11:26 < arubi> /maybe/ the concern is about recovering "2nd generation" tweaks\privkeys ?  like ones created by a previous snicker?  I still think our original ecdh method works, since the very first snicker must have come from privkey native to the master xprv
11:32 < arubi> yea gonna stop assuming and read through some of the thread now (I actually put the work laptop aside :) )
11:44 < arubi> okay waxwing I read up until your "Just to chime in" email and I realize that there was never a concern about recovering from xprv.  sorry for assuming so much :)  the watch-only thing is really interesting
12:06 < arubi> like..  really interesting.  can we make use of the chain code somehow to encode stuff into the signature that only the watch-only wallet can recover?  maybe both spending and watchonly wallets need to be smarter than just an xprv and xpub, say we had a second, secret, shared bip32 seed that both wallets shared (but isn't usable as spending keys)
12:08 < arubi> maybe not have bip32 derivation at all, but some other scheme that allows this kind of communicating of 32 byte values through signatures while keeping the "spending" and "watch-only" properties
12:10 < arubi> actually it would be much nicer to communicate the tweak*G instead of the tweak itself
12:22 < waxwing> arubi, re: chain code and other customisations, i don't get how you can get around the problem that whatever the tweak is, the proposer needs to know it too (assuming no interactivity, they had to generate tweak*G)
12:22 < waxwing> re your last part 'communicate tweak*G instead of tweak' yeah that could make sense for watch-only monitoring as you're by definition never going to be signing on the key, there.
12:23 < waxwing> although, huh, that does involve trusting that what you're being sent is true .. whereas if you send the tweak, it's verifiable that the xpub owner, owns that as well.
12:23 < arubi> waxwing, right I do mean we should keep ecdh for tweaks, but wondering if a secret 32 byte value shared by spending and watchonly wallets could help somehow for encoding 'c', but then yea it would be much better to encode c*G
12:24 < arubi> hmm, that's a good point..
12:26 < waxwing> yeah i can't see much value in going tweak*G there, since even from the privacy angle, if anyone can see that they can deduce which coinjoin outputs are owned, that's true whether it's the scalar or the curve point.
12:27 < waxwing> on your overall general point that it might be possible to hide in transactions, and have another secret or set of secrets to allow that encoding, i dunno, maybe, i could imagine it being possible (but complex).
12:28 < arubi> yea I mean we could just go scorched earth and encrypt the data in op return to some shared xprv
12:28 < arubi> but I /really/ want it to be in the signature :)
12:29 < arubi> and fwiw op return was used by stealth addresses kinda the same right
12:31 < arubi> (brb)
12:32 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has joined #joinmarket     
12:33 < waxwing> so wait, is the idea something like: take secrets on another bip32 branch (privkeys .. or do something special to make double sure never use those keys for coins), then use those to encrypt the tweaks for snicker txs and add them into signature nonces a la sign to contract .. i dunno it's cool but i don't see it without having privkey access due to the way nonces (determ.) work.
12:34 < waxwing> i mean for sure one can always communicate secrets with OP return, it's not nuts at least, even if it's not exactly desirable.
12:35 < waxwing> i don't like giving up at least the possibility that snicker coinjoins may be created that don't have an unambiguous flag of what they are (obv for now they'd be identifiable, but that's just for now).
12:44 < arubi> so what about, a snicker output is a 2-of-2 multisig.
12:44 < arubi> first pubkey is (d+c)*G (normal snicker tweaked pubkey), and the second key is like (c/x)*G, where 'x' is some sibling private bip32 path that the spending and watchonly (WO ?) wallets share
12:44 < arubi> now a WO wallet sees a spend from a pubkey it knows about, it looks at the second pubkey on the 2-of-2, and multiplies by 'x' which it knows about.
12:44 < arubi> the result is c*G.  it adds this c*G to the original pubkey and checks if the first pubkey of the 2-of-2 matches.
12:44 < arubi> if it does, then I think that's kinda authenticated?
12:45 < arubi> also wrt 2-of-2 outputs as a means of communication, it also kinda looks like LN I think? :)
12:48 < arubi> well yea it's pretty involved, not even sure if it works for chain snicker spends yet
12:52 < waxwing> belcher, AlexCato kristapsk others: my feeling is to merge #359 in ~two days after a squash. i will still be doing some minor tests.
12:53 < belcher> sounds good
12:53 < belcher> ill give the code another read
12:53 < waxwing> it was funny i was talking to ghost43 in berlin and he's doing PSBT right now in Electrum (which we will conveniently steal, perhaps!), and it's the same there: he knows that he has to merge it fast after it's finished since it touches almost everything in the codebase .. :)
12:53 < waxwing> well he said it does, on reflection that sounds a little weird, i guess it just depends.
12:54 < kristapsk> waxwing, there is some issue with my mainnet yg, but as I mentioned just in github, can't promise tests in coming days, I'm in Berlin hotel currently, with family
12:54 < waxwing> kristapsk, ok gotcha no worries, we'll just continue.
12:55 < kristapsk> maybe it could be merged, but probably not release new version yet, I really want to look at that, as it's reproducable, just need to find time for that
12:56 < belcher> what could psbt be used in joinmarket for?
12:56 < waxwing> well but it's not reproducible by 2 or 3 other people; if i could reproduce it i would certainly not ignore it :)
12:56 < waxwing> but what you said at the time suggested you weren't running the actual commit.
12:56 < waxwing> belcher, my immediate desire was for snicker (and only because it defines it as a compatibility standard, not because it needs it)
12:56 < belcher> aha     
12:57 < waxwing> otherwise yeah i wouldn't be that bothered about PSBT for now at least
12:57 < belcher> i thought also offline signing of coinjoins by takers? (if you do it fast enough)
12:57 < waxwing> it's not a priority but we could still have it anyway
12:57 < waxwing> yeah for that maybe, although like you say, can't see it happening today
12:57 < kristapsk> I guess it could be related that it's old wallet, running for years
12:57 < belcher> it used to be possible back in pre-clientserver days but i dont think anyone used it
12:58 < waxwing> kristapsk, well; could be; i have a lot of troubles with that. also bear in mind i fixed a bug that belcher found in syncing, it's conceivable that it's related. but i don't want to start a discussion without knowing what commit you were running on. and the specific functionality you were testing has been tested by others.
12:59 < waxwing> 'i have a lot of troubles with that' = myself and others do sometimes run into issues with really massive wallets (multiple thousands per mixdepth, say). better to switch after a thousand or two imo.
13:00 < kristapsk> hmm, ok, I could probably try re-test it again tonight, at least just with wallet-tool and see is it still reproducable
13:00 < kristapsk> with latest commits
13:02 < waxwing> belcher, i think that fell out of my memory but yeah now you say it: an option to not broadcast? didn't someone recently re-propose it? perhaps it was kristapsk ?
13:02 < waxwing> i know there were several PRs recently, was that one of them?
13:02 < waxwing> that would indeed fit with a hardware wallet and/or cold sign and hence yeah PSBT could fit there. but we're not quite there eh.
13:04 < kristapsk> waxwing, yes, I have PR for that (#414), it allows manual LN channel opening with JM coinjoins with c-lightning (tried to do that in the weekend, but fucked things up little bit, bugs in JM correct tx size calulation were part of it)
13:04 < belcher> waxwing yeah, although this one was called something like create-raw-coinjoin.py and it gave you a raw transaction sans signature which you had to move to the cold storage computer and run bitcoin core's signrawtransaction RPCs on and then broadcast
13:05 < belcher> also you had to provide a privkey for one of your UTXOs for the encryption (but other presumably larger UTXOs could be kept offline)
13:05 < waxwing> kristapsk, yeah i will def look into that fee calculation bug, i'm sure it needs to be changed, i think because it accepts estimation, when in certain cases it should be exact.
13:05 < kristapsk> my problem was that for a specific node long time ago I had set lower minrelaytxfee, so it now things that tx is ok, but other nodes don't relay it
13:06 < waxwing> belcher, oh, so i actually *had* forgotten. probably i never really read it carefully. ofc we can just add it back. Takers can use cold signing, in particular now we use BIP49 and BIP39 it may be more practical.
13:08 < belcher> we could but i doubt anyone ever used it
13:08 < belcher> it was like patientsendpayment.py which broke and nobody noticed for months
13:10 < waxwing> i think if it could be hooked up to a coldcard or trezor or ledger, they well might.
13:10 < waxwing> not sure all the details. i know coldcard has psbt support but, who knows. for the future :)
13:11 < belcher> interesting idea
13:11 < arubi> oh disregard what I said about 2-of-2, sorry, not sure what I was initially thinking at all :)
13:13 < kristapsk> waxwing, just checked for #359, still shows 0 balance
13:13 < kristapsk> checked out code in different directory, ran install.sh, copied joinmarket.cfg and wallets, ran wallet-tool.py summary
13:13 < waxwing> kristapsk, right; are you recovering? or is it a situation where it shows up fine with current master? i guess the latter?
13:13 < waxwing> i see, this is interesting. can you run again with --recoversync ?
13:14 < waxwing> hmm don't think that makes sense but try anyway.
13:14 < belcher> have you tried increasing the gap limit?
13:14 < waxwing> belcher, i thought that, but he said he's running with same config set up, so if it works in master ...
13:15 < kristapsk> it's ok with current master, yg is running against the same core wallet with current master fine
13:15 < kristapsk> also, wallet-tool shows different results, of course
13:17 < kristapsk> I will re-run with DEBUG loglevel and wallet-tool default method, to see which addresses it shows as recent ones
13:17 < waxwing> kristapsk, did you try --recoversync yet?
13:17 < kristapsk> nop, will try
13:18 < waxwing> as i said, it makes no sense that it would change, but it would make a more exact delta to current master.
13:18 < waxwing> (since default changed to fast)
13:20 < waxwing> i mean logically it sounds like a gap problem, but how would that have changed ...
13:22 < kristapsk> it shows correctly, for example, 0'/0'/0/37 for mixdepth 0 as first address
13:23 < kristapsk> but all funds are currently at INTERNAL addresses
13:23 < kristapsk> it does not show them at all
13:23 < kristapsk> will try --recoversync
13:25 < kristapsk> so, it looks everything is ok with 49'/0'/X/0/X, but not 49'/0'/X/1/X
13:27 < waxwing> after recoversync, you mean? and, is it still like you said, that it simply doesn't show any INTERNAL? (1/X)? that would be consistent with it just not finding the coins. (for what reason we haven't yet figured out).
13:28 < kristapsk> --recoversync does not change anything, the same result
13:28 < waxwing> hmm i have a slightly obscure theory. copy the wallet.jmdat file across again (from where it works, in your original directory, to the new wallets/), and then try again, but this time do it with --recoversync first.
13:30 < waxwing> hmm no don't think that theory makes sense. doesn't hurt to try of course, but i doubt it will change.
13:33 < kristapsk> waxwing, no, that doesn't help
13:33 -!- belcher [~belcher@unaffiliated/belcher] has quit [Quit: Leaving]     
13:35 < kristapsk>  got used indices: {0: [37, 945], ... <- these numbers in debug output when doing recoversync itself are correct
13:35 < waxwing> thanks.
13:36 < waxwing> this is 3a60.. right
13:37 < waxwing> you have rpc_wallet_file set i assume? (and therefore same in both)
13:37 < kristapsk> yes, 3a60c6759a1c4db3c29f711517d9516a2bfc8758
13:37 < kristapsk> no, i haven't, there is only single default wallet.dat in that node
13:38 < waxwing> hmm. i wonder if that could in any way be relevant; because behaviour is in some way matching what you'd expect if addresses were imported, but not in correct wallet?
13:39 < kristapsk> I can test that, fortunatelly there listwallets returns "wallet.dat", not "", so I can actually set it in joinmarket.cfg :)
13:40 < waxwing> or .. no not really, scratch that. the sync_addresses is checking for imports, if they're not there you would get different behaviour.
13:40 < waxwing> how many (if any) times do you see 'additional iteration required' in each case?
13:41 < waxwing> (in successful and unsuccessful cases/directories)
13:42 < kristapsk> with recoversync - once, when failed
13:42 < kristapsk> with failed I mean - zero balance as a result
13:44 < waxwing> and in the successful case (old dir, master)?
13:45 < kristapsk> hmm, I noticed one difference in log output with current master vs walletservice in --recoversync (should be same right?) - there aren't any listtransactions rpc logs in the second case
13:46 < kristapsk> in successful case the same, single "additional iteration required"
13:46 < waxwing> kristapsk, oh one thing - are you running default (display), not summary? could you fix on that to start with, as there may well still be a bug in summary, but want to get main display method working first.
13:47 < kristapsk> yes, I run default, did summary only for first test, default gives more info
13:48 < waxwing> ok good thanks
13:50 < waxwing> re: 'listtransactions' i added it to the quiet list in the new version at some point. it was spammy for some reason.
13:52 < waxwing> kristapsk, one more data point that could help me: could you run joinmarket-qt.py and load the wallet there, adn then after the sync, take a look at the Coins tab.
13:52 < waxwing> Or easier: just do showutxos method and see if the coins are there. then it would be some kind of weird display error.
13:52 < waxwing> also unlikely but i guess `showutxos` doesn't take time really.
13:54 < kristapsk> waxwing, harder, it's remote machine, I'm ssh'ing to it, could try wallet recover with Qt on laptop only, but that could be something completely different
13:54 < waxwing> right forget Qt, see the rest, just showutxos, should be nothing.
13:54 < kristapsk> yeah, ok, will try
13:56 < kristapsk> showutxos output is very compact :)
13:56 < kristapsk> {}     
13:57 < waxwing> yeah makes sense.
13:57 < waxwing> only one last thing, then leave it for now: try high gap limit, i don't know maybe 100.
13:59 < waxwing> i don't currently have a theory, it would need detailed debugging info in the two cases to see how they diverged. it seems like some kind of error where there isn't the right wallet label and, while the imports are all seen and so are the indices, but the transactions are not.
14:00 < waxwing> oh and just to check, you do see the message 'Wallet successfully synced.' near the end?
14:08 < kristapsk> yes, there is "Wallet successfully synced" msg when I do --recoversync
14:08 < kristapsk> will try gaplimit
14:16 < kristapsk> tried gap-limit 1000, still no change about internal addresses
14:17 < kristapsk> ok, I will deposit something to external address as a test, to be sure it is really issue with internal addresses
14:23 < waxwing> at this point it needs thinking and/or detailed reading of debug logs. after i've done something else tomorrow i will try to reproduce, can you give me approximate size of each index value? sounds like it's ~ 1K per index?
14:24 < waxwing> although even that i can't really pin down, that would make sense if only *some* of the coins were missing. that they are *all* missing suggests it's something else.
14:24 < waxwing> i had a moderate size (few hundred i guess on some indices) wallet that i started using on this new branch and didn't see this. this is really hard since i can't pin down any particular cause, all that can help here is really detailed logs until there is some clue.
14:25 < waxwing> AlexCato according to your reports no problems at the moment with this branch, do you have any other old wallets you can try syncing against?
14:27 < kristapsk> ahh, noticed additional detail
14:27 < kristapsk> there was actually one UTXO in external
14:27 < kristapsk> it's m/49'/0'/4'/0/3
14:27 < kristapsk> but new code shows external addresses only from m/49'/0'/4'/0/4
14:28 < kristapsk> initially I looked at first mixdepths and forgot to scroll to fourth one too
14:29 < waxwing> yes that is interesting but; it's a normal situation to show only addresses that are unused in the ext branch (for people to deposit); so the only thing wrong is, it sees 0/3 is used, but it doesn't see that it contains unspent coins. huh.
14:29 < waxwing> thanks for extra data point but i can't quite get the meaning yet.
14:46 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #joinmarket     
14:54 < AgoraRelay> [agora-irc/AlexCato] waxwing: i do. Checking right now.
14:55 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has quit [Remote host closed the connection]     
14:56 < AgoraRelay> [agora-irc/AlexCato] waxwing: works perfectly fine on a wallet which was in heavy use and is 2 or 3 years old
14:56 < AgoraRelay> [agora-irc/AlexCato] "works" as in i used the wallet-tool and it shows the correct balance and addresses
15:08 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has joined #joinmarket     
15:55 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Read error: Connection reset by peer]     
15:57 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
17:14 -!- AgoraRelay [~jmrelayfn@p5DE4AA06.dip0.t-ipconnect.de] has quit [Ping timeout: 240 seconds]     
17:32 -!- AgoraRelay [~jmrelayfn@p5DE4A9EF.dip0.t-ipconnect.de] has joined #joinmarket     
17:59 -!- lukedashjr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
17:59 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 245 seconds]     
18:05 -!- lukedashjr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 265 seconds]     
18:32 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
18:34 -!- viasil [~viasil@95.174.67.204] has quit [Ping timeout: 276 seconds]     
18:36 -!- viasil [~viasil@95.174.67.204] has joined #joinmarket     
18:42 -!- viasil [~viasil@95.174.67.204] has quit [Ping timeout: 265 seconds]     
18:43 -!- viasil [~viasil@95.174.67.204] has joined #joinmarket     
20:00 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has quit [Remote host closed the connection]     
20:00 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has joined #joinmarket     
20:06 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 276 seconds]     
20:12 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #joinmarket     
20:14 -!- viasil [~viasil@95.174.67.204] has quit [Ping timeout: 240 seconds]     
20:16 -!- viasil [~viasil@95.174.67.204] has joined #joinmarket     
20:26 -!- viasil [~viasil@95.174.67.204] has quit [Ping timeout: 265 seconds]     
20:28 -!- viasil [~viasil@95.174.67.204] has joined #joinmarket     
20:33 -!- Quitis [~Quitis@gateway/tor-sasl/quitis] has joined #joinmarket     
20:34 -!- Quitis [~Quitis@gateway/tor-sasl/quitis] has left #joinmarket []     
20:34 -!- viasil [~viasil@95.174.67.204] has quit [Ping timeout: 240 seconds]     
20:36 -!- viasil [~viasil@95.174.67.204] has joined #joinmarket     
20:54 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 268 seconds]