--- Log opened Tue Dec 18 00:00:56 2018
00:13 -!- kallewoof [~quassel@fp96f94c66.tkyc515.ap.nuro.jp] has quit [Remote host closed the connection]
01:00 -!- kallewoof [~quassel@fp96f94c66.tkyc515.ap.nuro.jp] has joined #rust-bitcoin
02:10 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has quit [Ping timeout: 260 seconds]
02:16 -!- midnightmagic [~midnightm@unaffiliated/midnightmagic] has joined #rust-bitcoin
02:19 < real_or_random> BlueMatt: I'm working on rust-minisketch (bindings) by the way. it's currently in a private repo, I'm just waiting for some upstream PRs to be fixed before I can polish it
02:32 -!- e4xit [~e4xit@cpc123762-trow7-2-0-cust7.18-1.cable.virginm.net] has quit [Ping timeout: 250 seconds]
02:33 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Remote host closed the connection]
02:33 -!- e4xit [~e4xit@cpc123762-trow7-2-0-cust7.18-1.cable.virginm.net] has joined #rust-bitcoin
03:04 -!- fabian [b9dc46c8@gateway/web/freenode/ip.185.220.70.200] has joined #rust-bitcoin
03:04 -!- fabian is now known as Guest32096
03:06 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #rust-bitcoin
03:18 -!- Guest32096 [b9dc46c8@gateway/web/freenode/ip.185.220.70.200] has quit [Quit: Page closed]
04:01 -!- belcher [~belcher@unaffiliated/belcher] has joined #rust-bitcoin
04:35 -!- e4xit [~e4xit@cpc123762-trow7-2-0-cust7.18-1.cable.virginm.net] has quit [Quit: quit]
04:46 -!- e4xit [~e4xit@cpc123762-trow7-2-0-cust7.18-1.cable.virginm.net] has joined #rust-bitcoin
07:45 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #rust-bitcoin
09:09 < BlueMatt> real_or_random: cool! that may be of interest to sdaftuar
09:10 < BlueMatt> but I dunno if he was gonna run with the network-sync-with-minisketch stuff or not
09:11 < BlueMatt> andytoshi: can I get concept ack on https://github.com/rust-bitcoin/bitcoin_hashes/pull/17 ?
11:18 < dongcarl> BlueMatt: Ah I needed that for integrating bitcoin_hashes into rust-bitcoin
11:33 < BlueMatt> oops, sorry dongcarl, i guess this makes 15 need a rebase
11:34 < dongcarl> Haha np, if you can do SHA512 as well that'd mirror all we have in `fuzz_utils` for `rust-bitcoin`
11:39 < BlueMatt> does rust-bitcoin have a sha512/ripemd fuzztarget?
11:39 < BlueMatt> I thought it was only sha256
11:54 < BlueMatt> dongcarl: https://github.com/rust-bitcoin/bitcoin_hashes/pull/18 :p
11:56 < dongcarl> BlueMatt: Haha I think we only needed SHA256 and SHA512, but doesn't hurt
11:56 < dongcarl> Does the logic need to match the existing one?
11:56 < dongcarl> Here: https://github.com/rust-bitcoin/rust-bitcoin/blob/master/src/fuzz_util/sha2.rs
11:57 < BlueMatt> preferably, yes
11:57 < BlueMatt> if it doesnt match it invalidates existing fuzz test cases
11:57 < BlueMatt> for rust-bitcoin its not a huge deal, cause the targets are simple
11:57 < BlueMatt> I dont really want to have to redo the rust-lightning test cases though
11:57 < BlueMatt> I believe that pr matches for sha512, though?
11:59  * dongcarl reading
12:01 < gmaxwell> BlueMatt: maybe we could find someone to talk into doing go bindings for minisketch, since that also would be a barrier for getting lnd to use it?
12:04 < BlueMatt> gmaxwell: I dont think we're anywhere near that point yet
12:04 < BlueMatt> *first* someone should figure out if it makes any sense in this context :p
12:09 < gmaxwell> BlueMatt: at least from what roastbeef told me it probably does (that a lot of bandwidth is spent learning things you already know from other peers).
12:10 < gmaxwell> It would just be sad if "omg I don't want to implement minisketch myself" were a reason to not use it, assuming it did make sense.
12:10 < BlueMatt> gmaxwell: I think yes, there are some contexts where it makes sense, but there are also multiple contexts that matter, and needs a bunch of investigation to see which ones make sense and how to integrate it in them
12:10 < BlueMatt> I dont think that'll be the reason, its better than fucking zlib
12:10 < gmaxwell> omg zlib die.
12:11 < gmaxwell> BlueMatt: minisketch todo does have a item on it for adding an entropy coder for efficienctly sending sets.
12:11 < dongcarl> We should talk to beef about this, not sure which IRC channel is appropriate tho
12:11 < gmaxwell> BlueMatt: because jesus christ even if retcon isn't a gain, ... zlib wtf.
12:11 < gmaxwell> (esp consider the 1.5 decades of vulns in libzlib, somehow...)
12:12 < BlueMatt> yes, I'm definitey not gonna integrate the zlib-based shit in rust-lightning
12:12 < BlueMatt> gmaxwell: I was trying to get sdaftuar to go do simulations but he's busy with other stuff
12:12 < gmaxwell> (something about dictionary based compressors seems to attract vulns)
12:12 < BlueMatt> gmaxwell: I'm happy to point you to how to write a dead-simple simulation framework and the relevant protocol bits if you want to go play :p
12:13 < BlueMatt> gmaxwell: I presume you saw rusty's post?
12:13 < gmaxwell> No.
12:13 < gmaxwell> BlueMatt: well what you should probably just do is take some real lightning nodes and istrument them to record how many things they get told about that they already now....
12:13 < gmaxwell> (simulation is important too, but getting figures from production is probably better for estimating potential savings)
12:14 < BlueMatt> gmaxwell: yes, that was my point
12:14 < BlueMatt> I'll point you to where in the code to shove instrumenters :p
12:15 < gmaxwell> yea, well that would require me to actually use lightning. I'm happy for other people to do that. At least until I get around to making my EV charger at home take lightning.
12:15 < BlueMatt> https://lists.linuxfoundation.org/pipermail/lightning-dev/2018-December/001741.html
12:16 < BlueMatt> heh, I figured you'd want a rust project
12:18 < BlueMatt> gmaxwell: re: volatile-based fixed_time_eq_slice function, godbolt thinks its right, only question is the final cmp TOTAL_XOR'D_OR'd_BYTE, 0
12:18 < BlueMatt> should I bit-twiddle that down to one bit to make sure that instruction is always only comparing one bit
12:18 < BlueMatt> or is it not worth it?
12:20 < BlueMatt> it does do a bunch of loop unrolling/branching based on input length, but thats fine
12:20 < gmaxwell> for(i=1;i<=16;i<<=1;) x |= x>>i;
12:20 < gmaxwell> or like that but correct.
12:20 < BlueMatt> yea, ok, was just curious if that was worth it
12:21 < gmaxwell> probably worth it, it's a one liner and needs just loglog(max) shifts+ors.
12:22 < gmaxwell> since it would be sad if your final comparison gets turned into a branch when it could be avoided.
12:30 < BlueMatt> ok, done... https://godbolt.org/z/SnRzLa
12:33 < BlueMatt> grrr, it turns the final (t & 0b1) | ((t & 0b10) >> 1); into a cmp 3
12:35 < BlueMatt> ok, this one keeps the high bits till the last step and then masks them out https://godbolt.org/z/wyhjd8
12:35 < BlueMatt> better
12:37 < BlueMatt> and without shifts means more high bits but thats ok https://godbolt.org/z/mMbGQv
12:37 < BlueMatt> err, ands
12:37 < BlueMatt> andytoshi: do you want this to be in bitcoin_hashes?
12:40 < andytoshi> BlueMatt: yeah, i think that's a good place for it
12:40 < andytoshi> i wonder if it should be the default Eq implementation even
12:40 < BlueMatt> i mean it bangs L1 hard
12:40 < BlueMatt> but...maybe?
12:40 < andytoshi> i guess not, it'd make dealing with public data brutally slow..
12:40 < andytoshi> yeah
12:40 < andytoshi> it's a hard call
12:41 < BlueMatt> I mean...unclear?
12:41 < BlueMatt> I havent benchmarked it
12:41 < BlueMatt> not like l1 is that slow
12:41 < andytoshi> well, i would expect a memcmp of 32 bytes to take like one cycle
12:41 < BlueMatt> heh well that this will not
12:41 < andytoshi> so doing anything at all is going to be slow relative to that
12:42 < andytoshi> i think it's worth benchmarking
12:43 < andytoshi> if it's less than 10% the cost of doing a hash compression function, say, we might as well make it the default
12:43 < BlueMatt> agreed
12:43 < BlueMatt> I mean I'd say like 50%, but ok
12:43 < BlueMatt> I mean it may be that the reason this is not used by the other rust constant-time-comparison crates is that its much slower
12:44 < BlueMatt> cause its kinda the "normal" way to do this, at least in the C world
12:44 < andytoshi> heh, yeah, i'd accept 50%
12:44 < andytoshi> right
12:52 < gmaxwell> careful, really awful performance is created by producing small performance leaks everywhere. :P  (then again, you're using a language that doesn't make the stack all the useful, so you've already failed in that respect. :P )
13:00 < BlueMatt> andytoshi: ok, I pr'd it as-is, you're welcome to go benchmark it but I'm not in a rush to do that and dont have a bench-able thing handy right now
13:10 < BlueMatt> few easy-merge prs at https://godbolt.org/z/mMbGQv :p
13:10 < BlueMatt> errr https://github.com/rust-bitcoin/bitcoin_hashes/pulls
13:27 < BlueMatt> andytoshi: heh, yea, its slloooowwww
13:27 < BlueMatt> test cmp::benches::bench_32b_constant_time_cmp ... bench:          28 ns/iter (+/- 0)
13:27 < BlueMatt> test cmp::benches::bench_32b_slice_cmp         ... bench:           0 ns/iter (+/- 0)
13:27 < BlueMatt> test cmp::benches::bench_64b_constant_time_cmp ... bench:          62 ns/iter (+/- 2)
13:27 < BlueMatt> test cmp::benches::bench_64b_slice_cmp         ... bench:           1 ns/iter (+/- 0)
13:29 < BlueMatt> could probably speed it up with less reads/writes (eg use usize instead of u8s everywhere), but tbh i think its fine for now
13:29 < BlueMatt> its like 50x slowdown, but whatever
13:36 < BlueMatt> heh, though tbh its almost probably not even worrying about constant-time comparison for 32 byte strings, fucking syscall latency will hide it :/
13:59 < andytoshi> heh, damn, 30 ns is pretty-much right on my "10%" dividing line .. like the worst possible result
14:00 < andytoshi> i'm leaning toward doing it because i _always_ forget to use constant-time comparisons for hash functions when i need them, since i'm almost always hashing public data anyway..
14:01 < andytoshi> and like, what kind of applications involve shitloads of hash function comparisons anyway?
14:01 < andytoshi> stuff like password verification that needs const-time..
14:57 -!- michaels_ [~michaelsd@38.126.31.226] has joined #rust-bitcoin
14:59 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Ping timeout: 240 seconds]
15:08 -!- michaels_ [~michaelsd@38.126.31.226] has quit [Remote host closed the connection]
16:55 < sgeisler> andytoshi, stevenroose: I implemented a minimal HTTP client for bitcoind: https://github.com/apoelstra/rust-jsonrpc/pull/22
16:55 < sgeisler> It still needs some testing but works for me so far.
16:57 < sgeisler> It supports a minimal subset of HTTP and will accept a superset of HTTP as response, but we basically don't care as long as it can correctly fetch JSON from a bitcoind node imo.
17:06 < sgeisler> I don't know if it's worth it but one could also think about a not so simple one that keeps a connection pool, but local TCP connection probably don't have that huge connecton overhead compared to the time request processing takes.
17:39 < ariard> BlueMatt: bookmarked minisketch and rusty protocol scratch for #57 network-sync, will stuck on bolt 5 for now, the whole ChannelMonitor thing still needs more care :)
17:44 -!- takinbo [sid19838@gateway/web/irccloud.com/x-vyuvelisdlbcrdve] has joined #rust-bitcoin
18:16 -!- e4xit [~e4xit@cpc123762-trow7-2-0-cust7.18-1.cable.virginm.net] has quit [Ping timeout: 240 seconds]
20:12 -!- grubles [~grubles@unaffiliated/grubles] has quit [Quit: Leaving]
21:54 -!- grubles [~grubles@unaffiliated/grubles] has joined #rust-bitcoin
23:01 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Remote host closed the connection]
23:37 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #rust-bitcoin
23:49 < dpc> sgeisler: Someone might want to use jsonrpc to a node over network (example in an isolated network/vpn or over some tunnel). However if anyone needs thread-pool they can probably do it themselves. As long as it's documented somewhere, it should be fine.
--- Log closed Wed Dec 19 00:00:56 2018