--- Day changed Thu Jul 12 2018
00:51 -!- undeath [~undeath@unaffiliated/undeath] has joined #joinmarket     
01:33 -!- undeath [~undeath@unaffiliated/undeath] has quit [Quit: WeeChat 2.1]     
03:04 -!- belcher [~belcher@unaffiliated/belcher] has joined #joinmarket     
06:01 -!- undeath [~undeath@unaffiliated/undeath] has joined #joinmarket     
06:19 < waxwing> undeath, testing your PR, i find that if i run another test e.g. jmclient/test/test_taker.py it runs fine, but if i run your test_coinjoin.py it doesn't seem to trigger the actions in conftest.py and doesn't start the regtest blockchain (which needn't be a problem except load_program_config does a test RPC call getblockchaininfo inside it which fails)
06:20 < undeath> that's weird
06:21 < undeath> shouldn't pytest load that file before every test?
06:21 < waxwing> yes i thought just importing pytest ought to be enough? not sure. i can't see what's different.
06:22 < waxwing> if you do something like `py.test --nirc=2 --btcroot=.. --btcpwd=123456abcdef --btcconf=.. jmclient/test/test_coinjoin.py you don't get any error?
06:24 < waxwing> maybe it's because you called it setup(), let me try that
06:24 < undeath> I only ran the test on the new-wallet branch with a long-running regtest
06:24 < waxwing> yep changed it to "xsetup" and now it's running :)
06:24 < undeath> oh, wow :/
06:24 < undeath> thank you for figuring that out!
06:24 < waxwing> oh but now it fails differently
06:25 < undeath> lol     
06:25 < waxwing> it's running the conftest.py code so it's loading the blockchain but something's wrong still
06:25 < undeath> did you rename it in both places?
06:25 < undeath> test_coinjoin argument and function name itself
06:25 < waxwing> no, lol
06:26 < waxwing> it's running now. seems to be loading a whole bunch of wallets?
06:26 < undeath> should be creating 4 wallets
06:26 < waxwing> ah, then it fails on the feerate thing. should we get that sorted now, before doing this?
06:26 < undeath> ah, true
06:27 < waxwing> about setup(), that was pure intuition from seeing the stupid things pytest usually does :) it has weird rules.
06:27 < undeath> it could at least warn about that :/
06:27 < waxwing> right. there are docs but they're perhaps not what they could be.
06:28 < waxwing> gonna pull in that feerate PR now. do you think it's right as-is?
06:28 < undeath> did test_commands run with that change?
06:28 < undeath> I haven't run it locally, just stared a while at the code
06:28 < waxwing> oh sorry already switched out :) will come back to that. want to get the feerate thing working if it's a block.
06:29 < waxwing> you also mentioned 100000 earlier, i agree, let's switch to 10k
06:29 < undeath> should I just change the pr?
06:31 < waxwing> yeah why not; and squash, thanks
06:31 < undeath> yep, will do
06:31 < waxwing> testing it now
06:31 < undeath> squash test_coinjoin pr, too?
06:32 < waxwing> no big deal, it's fine. just if it's very small like the feerate thing better not to add commits i think.
06:32 < undeath> second commit is refactoring of the first
06:33 < waxwing> i agree with keeping the commits separate while under review, i also agree it's good to keep it small (ideally 1) when finished, but don't want to be ultra-strict and annoying about it.
06:34 < waxwing> so in the case where the rpc doesn't return a feerate, it definitely still returns a dict right?
06:34 < waxwing> yeah must be otherwise you would have noticed
06:35 < undeath> yes     
06:35 < undeath> the "errors" key is still a part of the legitimate response
06:35 < undeath> either one should be present
06:36 < waxwing> right. it's fine then. switch to 10k and i'll merge it.
06:36 < undeath> already force-pushed
06:37 < waxwing> k, thx, when i've finished clacking keys for that, will switch back to the coinjoin test
06:37 < undeath> thank you!
06:40 -!- GitHub176 [GitHub176@gateway/service/github.com/x-xautmlzzjxyzbrxl] has joined #joinmarket     
06:40 < GitHub176> [joinmarket-clientserver] AdamISZ pushed 2 new commits to master: https://git.io/fNLwS
06:40 < GitHub176> joinmarket-clientserver/master fc0977c undeath: fix error condition with estimate fee
06:40 < GitHub176> joinmarket-clientserver/master 39eba7a AdamISZ: Merge #161: fix estimatesmartfee failing...
06:40 -!- GitHub176 [GitHub176@gateway/service/github.com/x-xautmlzzjxyzbrxl] has left #joinmarket []     
06:40 -!- GitHub11 [GitHub11@gateway/service/github.com/x-smvtksihldcomqgm] has joined #joinmarket     
06:40 < GitHub11> [joinmarket-clientserver] AdamISZ closed pull request #161: fix estimatesmartfee failing (master...fix_estimatesmartfee_regtest) https://git.io/fNT8k
06:40 -!- GitHub11 [GitHub11@gateway/service/github.com/x-smvtksihldcomqgm] has left #joinmarket []     
06:45 < waxwing> undeath, so do you need to rebase 160 or not?
06:46 < undeath> I can rebase it, that way we will see if travis succeeds it
06:46 < waxwing> i'm confused about this, if the changes are independent in code, there'll be no problem merging. so i should just rebase for my testing?
06:46 < undeath> other than that you should be able to just merge it
06:46 < undeath> because there are no merge conflicts
06:46 < waxwing> oh like merge into my local master?
06:46 < undeath> yeah
06:47 < waxwing> got it
06:47 < waxwing> you need to update the function name don't forget :)
06:47 < undeath> ah, yes
06:49 < waxwing> k, seems to work after that. creates tx, watcher loop and stops. no errors so tx must be valid.
06:49 < undeath> yep, that's what I see locally :)
06:49 < undeath> does test_commands run now?
06:49 < waxwing> i'll run the whole suite with that added to see
06:50 < undeath> weird thing is that test doesn't fail for me locally in the new-wallet branch
06:52 < waxwing> last update passed travis OK: https://travis-ci.org/JoinMarket-Org/joinmarket-clientserver/builds/403121937?utm_source=github_status&utm_medium=notification
06:52 < waxwing> so it's not going to be something where the travis env has changed or anythin
06:53 < waxwing> oh, so i reproduced it locally: test_commands failed. nothing else failed.
06:53 < waxwing> i wonder if some kind of naming conflict could cause that? i'll try just those two tests.
06:55 < undeath> just force-pushed with #161 merged, but it'll probably end up the same way as your local test
06:56 < waxwing> test_commands only: ok, test_coinjoin only: ok, both together (and nothing else); fails.
06:57 < waxwing> failures: "NOTE: Incompatible Exception Representation, displaying natively:" .. followed by ... stuff
06:57 < undeath> yeah. it fails somewhere deep in twisted
06:57 < waxwing> ah. i think it might be the tx_watcher loop.
06:58 < undeath> something along those lines was what I thought, too
06:58 < waxwing> the regtestbitcoincoreinterface is still watching a tx, for some reason, and somehow that doesn't play nice with how that weird twisted test setup shuts down. that's my guess.
06:58 < waxwing> "DirtyReactorAggregateError: Reactor was unclean."
06:58 < undeath> I could just monkeypatch that method
06:58 < waxwing> there's a looping call. i can't remember the rules now.
06:58 < waxwing> yeah might be easiest.
06:59 < waxwing> i guess in general that's something to pay attention to: we have the tests share a btccore interface and can leave stuff dangling in there.
07:00 < waxwing> the fact that twisted is totally weird is not necessarily the point (arguable...)
07:00 < undeath> are you sure it's the tx watcher?
07:00 < undeath> maker.py:27 self.sync_wait_loop = task.LoopingCall(self.try_to_create_my_orders)
07:00 < undeath> that's a possible candidate, too
07:01 < waxwing> hmm yes, it's a bit hairy that. the reasoning is ... somewhere. had to have the maker wait for synchronisation to complete.
07:03 < waxwing> it seems this is specific to twisted.trial
07:04 < waxwing> well, obviously. you're allowed to shutdown the reactor whenever you like. what makes testing twisted a pain is, there's only allowed to be one, global reactor. you can't even in serial, stop and restart.
07:05 -!- belcher [~belcher@unaffiliated/belcher] has quit [Read error: Connection reset by peer]     
07:05 -!- belcher [~belcher@unaffiliated/belcher] has joined #joinmarket     
07:06 < undeath> can we just call reactor.stop() at the end of the test?
07:08 < waxwing> i can't remember how it works now. i'll look.
07:09 < waxwing> undeath, so, your test is not using twisted at all, am i right?
07:10 < undeath> the test itself isn't, no
07:10 < waxwing> if that's the case i guess any loops you trigger are in a 'wait' state until the reactor starts.
07:10 < waxwing> iirc the thing i had to do was, for some tests, use the twisted.trial module which was the only way i found of integrating twisted code into a pytest framework
07:11 < waxwing> you can't (as i said), just start the reactor, do something, stop it, and wait for the next test. so it was quite annoying in that sense.
07:11 < waxwing> this was all quite a while ago, i may have forgotten some crucial details.
07:11 < undeath> the reactor seems to get started somehow somewhere
07:11 < undeath> because maker.offerlist gets populated
07:11 < waxwing> i don't think so. you can make twisted calls but nothing happens until the reactor starts. i think.
07:12 < undeath> and that can only happen by twisted executing the try_to_create_my_orders method
07:13 < undeath> the test just initializes the maker and later accesses that attribute, so some magic must happen in between
07:16 < waxwing> in the non-test scenario, you only get the reactor running by calling ygmain() in yieldgenerator.py (for Makers)
07:17 < waxwing> but .. i take your points above, i'm still trying to figure it out :)
07:17 < undeath> twisted seems to be … twisted
07:18 < waxwing> it is, but this is a problem of twisted with testing, it's much less confusing for normal running.
07:21 < undeath> does test_client_protocol start reactor maybe?
07:21 < waxwing> yeah but hang on, i still can't see evidence that try_to_create_my_orders is being called? i put a print statement and it never showed up with -s
07:22 < undeath> lol, how does self.offerlist get populated?
07:23 < waxwing> oh yes it's there, sorry
07:24 < waxwing> so .. i'll look at the docs for twisted.trial. your theory seems to make perfect sense, at least, if what i said above is right: you can't have more than one reactor.run() call in a process ... i guess.
07:24 < waxwing> oh damn, no it doesn't, because i just ran test_coinjoin.py on its own! doh.
07:24 < undeath> :(     
07:24 < undeath> magic
07:25 < waxwing> this is weird, how did you start the reactor? you didn't call reactor.run directly, and you didn't call start_reactor in client_protocol.py (note e.g. in test/ygrunner.py)
07:26 < undeath> reactor is also running in other tests because pushtx uses reactor.callLater and tx do get pushed to bitcoind
07:27 < waxwing> right. so i guess it must be: py.test automatically starts it if it sees twisted.trial being used, something like that?
07:28 < waxwing> the reason it somehow passed me by is, i started writing the tests before it was all reactor based; originally the bitcoin/blockchain part was using threads for that stuff.
07:28 < waxwing> then i rewrote it so it was all without threads.
07:28 < undeath> oh, wait, no, it actually just calls BitcoinCoreInterface.pushtx with uses self.rpc
07:29 < waxwing> ok. well anyway as we have observed, the reactor is running :)
07:30 < undeath> yeah, some implicit start of reactor is the only explanation I can think of
07:30 < waxwing> i think we should quiet the test_commands test for now, and open an issue. that code is basically completely unchanging anyway. this is not going to be easy to iron out.
07:30 < undeath> https://docs.pytest.org/en/latest/faq.html#how-does-pytest-relate-to-twisted-s-trial
07:32 < undeath> tl;dr: pytest magic
07:32 < waxwing> yeah i didn't understand that.
07:39 < waxwing> worst comes to worst, we can after all create a completely separate test script which doesn't use pytest and uses twisted as intended. (in addition to the pytest one).
07:40 < undeath> according to the debug output it's Taker.unconfirm_callback and RegtestBitcoinCoreInterface.tx_watcher
07:40 < undeath> if I monkeypatch both the issue should be gone
07:42 < waxwing> it might be necessary. if i ignore test_command i get an error in test_daemon_protocol
07:42 < waxwing> arubi, where's the py.test command gone? i can't find it in install.sh
07:43 < waxwing> oh run_tests.sh sorry
07:44 < undeath> line 48
07:44 < waxwing> yeah i got it, i originally forgot it was that file
07:46 < undeath> is there a better way than monkeypatching the methods? like, cancel all delayedcalls in reactor?
07:47 < waxwing> lol that might be the biggest stack trace i've ever seen (in test_daemon)
07:48 < waxwing> undeath, i don't know. interesting thought. not sure if i've seen that method but bet it exists.
07:54 < waxwing> tx_watcher, also tx_network_timeout
07:55 < waxwing> i think we just have to modify the call to push and probably the maker at the point where it sends sig to stop it creating various watching loops
07:56 < undeath> looking through twisted docs, maybe we can get an iterator for all open delayedcalls and cancel them
07:56 < waxwing> but it also depends what you want to test; you could just make a new BlockchainInterface class (better than existing DummyBCI but still)
07:57 < waxwing> overwrite add_tx_notify so it does nothing
07:58 < waxwing> think i'll try that
07:59 < undeath> new class to overwrite pushtx from RegtestBlockchainInterface?
08:04 < undeath> https://twistedmatrix.com/documents/current/api/twisted.internet.base.ReactorBase.html#getDelayedCalls
08:04 < undeath> :)     
08:05 < waxwing> oh cool. and you can try calling cancel. but not sure where to do that? in a finalizer for the module scope fn perhaps
08:05 < undeath> yep     
08:05 < waxwing> trying out the other way fwiw
08:09 < undeath> seems to work :)
08:10 < waxwing> Your way? cool. I got:
08:10 < waxwing> DirtyReactorAggregateError: Reactor was unclean.
08:10 < waxwing> DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
08:10 < waxwing> <DelayedCall 0x7f3bfd5eb2d8 [2.7742497921s] called=0 cancelled=0 LoopingCall<5>(RegtestBitcoinCoreInterface.tickchain, *(), **{})()>
08:10 < waxwing> in other words the regtest tick function was annoying it.
08:10 < waxwing> all the other ones went though.
08:10 < undeath> i printed reactor.getDelayedCalls() before and after canceling them all in a for loop
08:10 < undeath> in teardown
08:10 < undeath> last log entry was an empty list :)
08:10 < waxwing> right, great. that's a good solution.
08:11 < waxwing> i wonder if some other test functions were implicitly depending on the tick function still running .. lol.
08:12 < waxwing> still completely amazed that the reactor is running when you just do jmclient/test/test_coinjoin. it must just go from even having twisted imports in the modules that get imported. or something. i'm sure the reactor isn't manually started anywhere.
08:13 < undeath> it's pytest magic
08:13 < undeath> just as written in their faq ;)
08:14 < undeath> pushed to pr
08:14 < undeath> lets see what travis says
08:20 < waxwing> fixed it here (whole suite passed)
08:20 < waxwing> i'll merge it assuming travis is ok too
08:24 < undeath> travis is happy!
08:25 -!- GitHub165 [GitHub165@gateway/service/github.com/x-qymhhtnwghiejdef] has joined #joinmarket     
08:25 < GitHub165> [joinmarket-clientserver] AdamISZ pushed 3 new commits to master: https://git.io/fNL9S
08:25 < GitHub165> joinmarket-clientserver/master 4524293 undeath: add basic test for coinjoins
08:25 < GitHub165> joinmarket-clientserver/master 2bfe080 undeath: add teardown code in test_coinjoin for clean reactor
08:25 < GitHub165> joinmarket-clientserver/master 2a55294 AdamISZ: Merge #160: add basic test for coinjoins...
08:25 -!- GitHub165 [GitHub165@gateway/service/github.com/x-qymhhtnwghiejdef] has left #joinmarket []     
08:25 -!- GitHub88 [GitHub88@gateway/service/github.com/x-ulyrmiactzhlujhk] has joined #joinmarket     
08:25 < GitHub88> [joinmarket-clientserver] AdamISZ closed pull request #160: add basic test for coinjoins (master...test_coinjoin) https://git.io/fNTZl
08:25 -!- GitHub88 [GitHub88@gateway/service/github.com/x-ulyrmiactzhlujhk] has left #joinmarket []     
08:32 -!- gmaxwell [gmaxwell@wikimedia/KatWalsh/x-0001] has quit [Ping timeout: 256 seconds]     
08:33 -!- gmaxwell [gmaxwell@mf4-xiph.osuosl.org] has joined #joinmarket     
08:33 -!- gmaxwell is now known as Guest81320     
09:40 -!- Guest81320 [gmaxwell@mf4-xiph.osuosl.org] has quit [Changing host]     
09:40 -!- Guest81320 [gmaxwell@wikimedia/KatWalsh/x-0001] has joined #joinmarket     
09:40 -!- Guest81320 is now known as gmaxwell     
09:41 -!- arubi [~ese168@gateway/tor-sasl/ese168] has quit [Remote host closed the connection]     
09:41 -!- arubi [~ese168@gateway/tor-sasl/ese168] has joined #joinmarket     
10:11 -!- undeath [~undeath@unaffiliated/undeath] has quit [Quit: WeeChat 2.1]     
11:35 -!- dx25 [~dx25@97-119-117-177.omah.qwest.net] has quit [Ping timeout: 264 seconds]     
12:58 -!- barfbaby [a2d82eb6@gateway/web/freenode/ip.162.216.46.182] has joined #joinmarket     
12:58 < barfbaby> How do I get coins out of the joinmarket wallet?
13:12 < waxwing> barfbaby, easiest way is to use `python sendpayment.py -N 0 -m (whatever mixdepth) walletname amount-in-sastoshis destination-address`
13:12 < waxwing> that sends without joins or IRC, to the destination-address
13:12 < waxwing> can only do it one mixdepth at a time.
13:12 < waxwing> it is also possible to load the wallet in trezor, ledger, electrum but that isn't great for privacy.
13:13 < waxwing> also samourai i think, and 1 or 2 others
13:13 < barfbaby> Thanks I'll give that a try.  Is that also the best way to move between depths?
13:13 < waxwing> well you can do that, or do a coinjoin (-N nonzeronumber); but moving between depths like that is generally counter to the purpose. think of them as separate accounts. better not to merge the coins.
13:14 < barfbaby> That makes sense
13:15 < waxwing> if you do use a coinjoin to move from one mixdepth to the other that's OK. that's after all how the tumbler works.
13:15 < barfbaby> "amount-in-sastoshis" is the amount to transfer?
13:16 < waxwing> the idea is that "connecting" one mixdepth (account) with another is something you're avoiding, except by doing it inside a coinjoin, which obfuscates which output is tied to your inputs.
13:16 < waxwing> amount in satoshis yes e.g. 0.1 btc enter 10000000
13:17 < waxwing> not sure anyone's that interested but i just calculated that the !auth message will send ~ 670 chars on the IRC, so it'll basically always be 2 messages.
13:19 < barfbaby> Is there any importance to the sequence of wallets?  Doyou need to add coins starting with 0 and go up?
13:19 < waxwing> barfbaby, good Q, and the answer's no
13:20 < waxwing> belcher, has expressed regret on calling them "mixdepths" because of this; they aren't depths in any sense
13:20 < waxwing> it's just that for the main algorithm "tumbler" we go through them one by one. but there's no ordering. when you offer coins for joins as a maker, you offer from any/all of them, whichever one happens to suit that particular transaction.
13:20 < barfbaby> Good to know.  I tried running yield generator, but it didn't seem like anyting happened.
13:21 < waxwing> you have to leave it running a long time, also i suggest using super-small fees (like 1-1000 satoshis, try `swabsoffer` see the top of the file , also try yg-privacyenhanced.py instead of -basic, although they're not that different)
13:22 < waxwing> "see the top of the file" = read the start of yg-privacyenhanced.py and follow the comment that tells you which numbers you can edit there.
13:25 < waxwing> also if you're impatient to see a coinjoin in action, you can always run `python sendpayment.py..` as mentioned above; i did one as a test about an hour ago and it worked fine .. it's pretty cheap.
13:25 < barfbaby> I waited atleast a week so I'm not that impatient
13:29 < waxwing> right :) actually i saw about 20 ish joins in the last 5-7 days (roughly) so it'll depend on fee but also on what range of sizes.
13:29 < waxwing> and it's still the case that activity is sporadic, anyway, not possible to predict.
13:31 -!- MaxSan [~user@185.9.19.107] has joined #joinmarket     
13:44 < barfbaby> What minimum amount of coins is needed?
14:04 -!- barfbaby [a2d82eb6@gateway/web/freenode/ip.162.216.46.182] has quit [Ping timeout: 252 seconds]     
14:05 < AgoraRelay> [agora-irc/CatoAlex] theres no real minimum. I've even seen smallish join with less than 0.01 btc lately. But most seem to be in the range 0.2 to 2 btc
14:06 < AgoraRelay> [agora-irc/CatoAlex] so if you have small fees and are offering more than 0.2 btc, you should see a coinjoin at least every other day or so
14:23 -!- barfbaby [a2d82eb6@gateway/web/freenode/ip.162.216.46.182] has joined #joinmarket     
14:23 < barfbaby> well shit...I tried a multiple join and my power went out and now all the coins are gone
14:26 < barfbaby> maybe they are back...I can see them in bitcoin core
14:49 < MaxSan> its either mixed or never left
14:49 < MaxSan> they dont just go
15:04 -!- belcher [~belcher@unaffiliated/belcher] has quit [Quit: Leaving]     
15:56 < grubles> >actually i saw about 20 ish joins in the last 5-7 days
15:56 < grubles> wow SWIM hasn't seen any in that amount of time >_>
16:20 -!- barfbaby [a2d82eb6@gateway/web/freenode/ip.162.216.46.182] has quit [Ping timeout: 252 seconds]     
16:59 -!- belcher_ [~user@unaffiliated/belcher] has quit [Ping timeout: 264 seconds]     
17:01 -!- belcher_ [~user@unaffiliated/belcher] has joined #joinmarket     
17:27 -!- AgoraRelay [~jmrelayfn@p5DE4AFA8.dip0.t-ipconnect.de] has quit [Ping timeout: 240 seconds]     
17:41 -!- AgoraRelay [~jmrelayfn@p5DE4AFB4.dip0.t-ipconnect.de] has joined #joinmarket     
20:42 -!- rdymac [uid31665@gateway/web/irccloud.com/x-bzvocenpwjjvoysi] has quit [Quit: Connection closed for inactivity]     
23:32 < waxwing> did you follow my suggestion to use negligible fees? check the orderbook for fees. if you're concerned you may be misconfigured sanity check by coming from Taker side and use -P to choose yourself. check IRC and check your bot is announcing. etc.
23:32 < waxwing> also what's SWIM
23:32 < waxwing> grubles, ^
23:49 -!- arubi [~ese168@gateway/tor-sasl/ese168] has quit [Ping timeout: 250 seconds]     
23:52 -!- arubi [~ese168@gateway/tor-sasl/ese168] has joined #joinmarket