--- Log opened Tue May 14 00:00:12 2019 00:09 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has joined #rust-bitcoin 00:11 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has quit [Ping timeout: 256 seconds] 00:40 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 00:41 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 00:45 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 246 seconds] 00:48 -!- TamasBlummer1 [~Thunderbi@p200300DD67196B856506B7C4D522E0CB.dip0.t-ipconnect.de] has joined #rust-bitcoin 00:50 -!- TamasBlummer [~Thunderbi@p200300DD67196B466506B7C4D522E0CB.dip0.t-ipconnect.de] has quit [Ping timeout: 264 seconds] 00:50 -!- TamasBlummer1 is now known as TamasBlummer 03:04 -!- mauz555 [~mauz555@2a01:e35:8ab1:dea0:20f1:7ad5:de01:c969] has joined #rust-bitcoin 03:13 < stevenroose> dpc: that sounds like it could be combined into a test that tests the rust-bitcoincore-rpc crate and all its methods 03:13 < stevenroose> we've been wanting something like that for a long time 03:14 < stevenroose> but yeah that would be integrated in the rust-bitcoincore-rpc CI tests, so perhaps not at easy to combine 03:39 < stevenroose> "Whether this private key should be serialized as compressed" in bitcoin::util::PrivateKey 03:47 -!- mauz555 [~mauz555@2a01:e35:8ab1:dea0:20f1:7ad5:de01:c969] has quit [] 04:20 -!- jtimon [~quassel@181.61.134.37.dynamic.jazztel.es] has joined #rust-bitcoin 05:50 < stevenroose> andytoshi: ping: https://github.com/rust-bitcoin/bitcoin_hashes/pull/39 05:50 < stevenroose> any blocker for that? 05:52 < stevenroose> I think we can merge sgeisler's #44 as well. 05:53 < stevenroose> Anything else we've been looking at for a new bitcoin_hashes release? 05:53 < stevenroose> It would be nice to get the AssetId stuff into rust-elements and do a rust-elements release with Address and AssetId. 06:01 < andytoshi> stevenroose: how are you going to make the taproot precomputed midstates? 06:02 < andytoshi> if the internal representation of Midstate is not public? 06:17 < stevenroose> andytoshi: have a [u8; 32] as a const and then wrap it with `from_inner` every time it's used. I believe we discussed that. That wrapping should be 0 overhead. 06:18 < stevenroose> (because from_inner is not a `const fn`) 06:20 < andytoshi> wrap it with from_inner, then from_mistate to get an engine? 06:20 < andytoshi> yeah that sounds fine actually 06:20 < andytoshi> i have to go now to speak at consensus. will be free in 3-4 hours hopefully. will test/merge then 06:22 < stevenroose> yeah. actually I can quickly try to implement that in the tagged hash PR I have open 06:26 < andytoshi> yeah, it'd be great if you could test the pr against rust-elements 06:34 < stevenroose> andytoshi: that has been done already.. 06:34 < stevenroose> and for taproot I realize that there is no from_midstate for engine yet. 06:36 < stevenroose> andytoshi: in fact, the sha256::HashEngine's internals are not [u8; 32], but [u32; 8] + usize.. so if we would do `HashEngine::from_midstate([u8; 32])` for the taproot hashes, that would still involve an additional convertion from [u8; 32] to [u32; 8] 06:42 < stevenroose> I don't really know the importance of the `length` variable in sha256::HashEngine.. nickler? 06:59 < andytoshi> it is critical to security 06:59 < andytoshi> and to correctness 07:00 < andytoshi> but i believe it should exist for every md hash; so we can do a `from_midstate` function that takes both a midstate and a length 07:01 < andytoshi> you can see in `Hash::from_engine` that we add the length to the end of the hashed data 07:33 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 07:40 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #rust-bitcoin 07:52 < stevenroose> andytoshi: if it's critical to security and correctness, why is it not part of the midstate? Can't the midstate be just `([u32; 8], usize)`? 07:57 < stevenroose> andytoshi: a quick reading of the code suggests the length is only used to check modulo BLOCK_SIZE, so it can be reset to 0 every BLOCK_SIZE. In fact, internally it could be just a u8 and store a number module BLOCK_SIZE, couldn't it? Not that usize overflow is a concern.. (I don't know if Rust is compiled for 16-bit processors? :D) 07:57 -!- jtimon [~quassel@181.61.134.37.dynamic.jazztel.es] has quit [Quit: gone] 07:59 < stevenroose> The thing is that if we can reduce the length to a single u8, the midstate is just [u8; 33] :) 07:59 < stevenroose> (Not that that's better than `([u8; 32], u8)` IMO. 08:20 < stevenroose> argh, I missed the `e.write_u64::(8 * data_len).unwrap();` 08:20 < stevenroose> sha256 commits to the total number of bits written before padding?? 09:28 < andytoshi> stevenroose: by definitions it's not part of the midstate 09:28 < andytoshi> and this is why it's not safe to use midstates 09:30 < andytoshi> also we absolutely write the entire length into the hash engine in `from_engine` 09:31 < andytoshi> not mod anythign 09:51 < stevenroose> yeah I realized that (see my "argh, .." message above). so from_midsate should take a midstate byte slice and a the length? That means that it can only be used for engines that have an empty buffer, i.e. when a number of bytes equal to a multiple of the block size has been processed, is that correct? 09:51 < stevenroose> For the tagged hashes, that's made for it 09:52 < stevenroose> For the tagged hashes, they are fine with that* 09:56 < andytoshi> correct 09:57 < andytoshi> hmm, actually the `midstate()` function on the midstate should probably be flushing the buffer.. 09:57 < andytoshi> ah actually that's not possible 09:58 < stevenroose> I guess we should add that to the documentation thenb 10:01 < stevenroose> andytoshi: only thing missing is a getter for the length. with the current `.midstate() -> MidState` and `from_midstate(MidState, usize)` API, the user is kinda required to remember the length. 10:02 < stevenroose> While in most (all?) legit cases of the midstate, the user should probably know the length to enter there, I'm not sure if it makes sense to require that. 10:02 < andytoshi> i think it's fine to require it for now 10:02 < andytoshi> like, i really want this to be hard to use 10:03 < andytoshi> and minimally sufficient for taproot and elements' fastmerklehash 10:10 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 10:11 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 10:15 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 268 seconds] 10:16 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 10:17 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 10:17 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 10:18 < stevenroose> for the compact blocks stuff, people mentioned that the siphash dependency could be made part of bitcoin_hashes. I'm looking at that, it doesn't seem like SipHash entirely fits in the traits. One example of what I bumped into is that Hash::engine() assumes engines have a default initial state, while SipHash takes 2 u64 keys. Also not sure it has a real midstate value, I'm finding that out. 10:22 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 245 seconds] 10:25 < andytoshi> i wouldn't be too heartbroken if siphash had to be its own thing 10:32 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has quit [Ping timeout: 256 seconds] 10:37 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 10:38 < stevenroose> I'm trying :) 10:47 < stevenroose> `the trait bound `siphash24::Hash: std::borrow::Borrow<[u8]>` is not satisfied` 10:47 < stevenroose> :| 10:48 < stevenroose> It doesn't make sense to have the Hash type for siphash be [u8; 8], right? instead of the u64 like it is in all implementations 10:49 < stevenroose> Actually it might, at least for compact blocks.. 10:49 < andytoshi> sure, it could be [u8; 8] 10:49 < andytoshi> i don't think everything else is u64 only 10:49 < andytoshi> i thought some were u32 11:02 < stevenroose> andytoshi: I noticed that the type name on HashEngine is `MidState` (note capital S). I named the sha256 type `Midstate`. Do you want me to rename it to have the capital S? 11:07 < andytoshi> hmmm 11:07 < andytoshi> Yea I guess you should 11:07 < andytoshi> best to be consistent 11:07 < andytoshi> though "Midstate" is more correct 12:32 -!- CubicEarth [~CubicEart@c-67-168-1-172.hsd1.wa.comcast.net] has quit [Ping timeout: 252 seconds] 12:32 < stevenroose> hmm... the MidState type is never used by users, though, so no one knows it's "less correct" :D we could just keep that hidden and keep the Midstate type that might be used externally as that :) 12:34 < andytoshi> what do you mean it's never used by users? 12:34 < andytoshi> is it not exposed? 12:40 -!- CubicEarth [~CubicEart@c-67-168-1-172.hsd1.wa.comcast.net] has joined #rust-bitcoin 12:48 < andytoshi> I'd like to implement display for TxOut 12:48 < andytoshi> any ideas what that should look like? 12:49 < andytoshi> right now in liquid logs there are lots of debug print like `TxOut { value: 99000000, script_pubkey: Script(OP_DUP OP_HASH160 OP_PUSHBYTES_20 b0fc5a05bd9b3ebc9e7101529d3fb6cb06081935 OP_EQUALVERIFY OP_CHECKSIG) },` which is super hard to read 12:50 < andytoshi> maybe hex_scriptpubkey/decimal_amount ? 12:50 < andytoshi> like, with a literal / separator 13:11 < stevenroose> hmm, I don't see how that is easier to read :p just a few characters less verbose. are you suggesting hex for the script because you don't care about the script? :D 13:13 < andytoshi> hex is much easier to read than that OP_whatever mess 13:13 < andytoshi> if you can't read Script as hex you should study more ;) 13:14 < andytoshi> also the amount could be a decimal amonut 13:14 < andytoshi> in bitcoin, rather than being in satoshi 13:50 < stevenroose> hmm, an alternative would be to overwrite Debug to something slightly less verbose: `TxOut{ X.XX BTC; }` 13:51 < stevenroose> the problem that I would see with a `X.XX BTC | ` Display is that you need to know the format to recognize it as a TxOut, which is not always obvious 13:52 < stevenroose> It's not like OutPoint where the format is somewhat standard : 13:52 < stevenroose> But well, that's a minor thing. I don't really have an opinion. I ported the SipHash24 impl to bitcoin_hashes and the test vectors don't work. Always nice. 13:55 < stevenroose> Ah, I see. Their unit tests make use of std::hash::Hasher 14:30 < stevenroose> https://github.com/rust-bitcoin/bitcoin_hashes/pull/46 14:31 < stevenroose> TamasBlummer: BIP158 also uses SipHash24, so that implementation should be sufficient. 14:33 < gmaxwell> andytoshi: IIRC sipahash is so fast that even function call dispatch is a meaningful overhead, I think we got a non-trival speedup on CB recovery by making a version that directly takes a 256 bit input. 14:33 < stevenroose> TamasBlummer: any plans to implement BIP158 in rust-bitcoin?? 14:39 < stevenroose> gmaxwell, andytoshi: so actually the siphash algorithm works on u64's so if we have the Hash type be [u8; 8], we do the conversion for nothing while might having to undo it. 14:39 < stevenroose> andytoshi: is it possible to implement Borrox<[u8]> on a u64 using unsafe? :D 14:40 < andytoshi> lol no 14:40 < andytoshi> please do not introduce endianness dependencies into these libraries 14:42 < stevenroose> good point :D is Borrow<[u8]> a hard requirement for the Hash trait? I mean we could have it implemented on all the other Hash implementations without needing it as a requirement for the Hash trait itself.. 14:44 < andytoshi> i don't understand how we would possibly use this without doing the conversion at some point 14:44 < andytoshi> are these not serialized or deserialized? 14:44 < stevenroose> for compact blocks they are 14:45 < andytoshi> without being serialized anywhere? 14:45 < stevenroose> no no I mean in compact blocks they are serialized 14:45 < stevenroose> I'm reading BIP158 to see what they do 14:46 < stevenroose> > The 64-bit SipHash outputs are then mapped uniformly over the desired range by multiplying with F and taking the top 64 bits of the 128-bit result. This algorithm is a faster alternative to modulo reduction, as it avoids the expensive division operation. 14:46 < stevenroose> so they do `return (siphash(k, item) * F) >> 64` (F is u64) 14:47 < andytoshi> okay, and what is done with that? 14:48 < stevenroose> and then for the golomb encoding they encode that differentially 14:48 < stevenroose> so they take the item_hash - last_item_hash 14:49 < andytoshi> okay 14:49 < andytoshi> so this is really really different from any other hash 14:49 < andytoshi> hashes can't be subtracted 14:49 < stevenroose> very confusing to me, differential encoding over something uniform makes no sense, right? so the multiplication with F would somehow make it non_uniform? 14:49 < andytoshi> if the intent is that people be subtracting siphashes then they should just do their own thing and not mess up the Hash trait for everyone else 14:50 < stevenroose> I agree. But I think for the purpose of using the u64 directly, we could just have a sip-specific method Hash::hash_to_u64 or something 14:51 < stevenroose> k, fine, I'll add a method for that use case and then we can keep the [u8; 8] as it is now 14:51 < andytoshi> ok, but then we're converting to [u8; 8] and then back to (u64, u64) for subtraction and then back to [u8; u8] to encode the result? 14:52 < andytoshi> just so SipHash can impl the Hash trait? 14:55 < stevenroose> no no I'll just make a method that returns the u64 and use that one to create the Hash in from_engine. 14:58 < andytoshi> oh i see 14:59 < gmaxwell> 14:49:10 < stevenroose> very confusing to me, differential encoding over 14:59 < gmaxwell> something uniform makes no sense, right? 14:59 < gmaxwell> it's not uniform, the values are sorted. It's communicating a _set_ 14:59 < stevenroose> gmaxwell: aaah, that makes sense! they communicate the sorted values indeed 15:00 < gmaxwell> the difference between the nearest uniform values are exponential. 15:00 < gmaxwell> and so the set is efficiently encoded at pretty close to the information theortic entropy. 15:01 < gmaxwell> (in fact the exact FP rate for the BIP158 is set to minimize the inefficiency of the coding scheme) 15:01 < stevenroose> andytoshi: I think the siphash PR looks OK now. BIP158 and BIP152 compatible. 15:02 < andytoshi> dope 15:02 < andytoshi> i can't review it tonight. possibly tomorrow, though i've got more interviews 15:03 < gmaxwell> (compact blocks isn't communicating a set, otherwise it would have done something similar) 15:08 < stevenroose> gmaxwell: it's communicating a list :) 15:10 < gmaxwell> kinda, duplicated entries just get ignored. 15:10 < gmaxwell> but avoiding that wasn't worth the complexity/cpu-time. 15:10 < stevenroose> andytoshi: I ran a super simple benchmark on the overhead and it's non-existent for hashing 1k bytes or 64k bytes 15:10 < andytoshi> of the conversion? 15:11 < andytoshi> yeah, on a LE machine i'm pretty sure it's compiled to nothing 15:11 < stevenroose> yeah the conversion 15:11 < stevenroose> hmm, yeah true, my machine doesn't really make sense for that 15:11 < gmaxwell> (if compact blocks was created without caring about CPU time, the hash for each entry would be salted by a small hash of the prior txids, so that dynamic programming could be used to resolve conflicts) 15:12 < stevenroose> I added the benchmarks to the PR, so they could be ran on whatever machine someone is curious about 15:13 < stevenroose> gmaxwell: what do you mean with "so that dynamic programming could be used to resolve conflicts"? 15:14 < gmaxwell> stevenroose: if every entry's hash depended on the txids in prior entries, then when you got one wrong, you'd notice that you'd get a ton of mismatches for all entries after it, and you could go back and try the other colliding entry. 15:16 < stevenroose> gmaxwell: ah that actually makes sense :D 15:16 < stevenroose> "small hash of the full txid instead of shortid, yeah that makes sense" 15:16 < stevenroose> (wrong quotes) 15:17 < gmaxwell> stevenroose: right, even without doing that, it would have been beter if each hash were salted by it's index. (so that repeated values wouldn't be automatic collisions), but again-- wasn't worth the added cpu time. 15:17 < gmaxwell> (I tried both of those constructions) 15:31 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has joined #rust-bitcoin 15:39 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection] 15:57 -!- windsok [~windsok@2604:a880:2:d0::1bda:1001] has joined #rust-bitcoin 15:57 -!- windsok [~windsok@2604:a880:2:d0::1bda:1001] has quit [Changing host] 15:57 -!- windsok [~windsok@unaffiliated/windsok] has joined #rust-bitcoin 16:02 -!- windsok [~windsok@unaffiliated/windsok] has quit [Client Quit] 16:06 -!- ghost43_ [~daer@gateway/tor-sasl/daer] has joined #rust-bitcoin 16:06 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Ping timeout: 256 seconds] 17:21 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 18:24 -!- ghost43_ is now known as ghost43 18:42 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 18:44 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has quit [Ping timeout: 256 seconds] 18:45 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Remote host closed the connection] 18:46 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #rust-bitcoin 18:59 -!- simian_za [~simian_za@195.159.29.126] has quit [Quit: Ping timeout (120 seconds)] 18:59 -!- CjS77 [~caylemeis@195.159.29.126] has quit [Read error: Connection reset by peer] 18:59 -!- simian_za [~simian_za@195.159.29.126] has joined #rust-bitcoin 18:59 -!- CjS77 [~caylemeis@195.159.29.126] has joined #rust-bitcoin 19:59 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 20:00 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 20:04 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 258 seconds] 20:16 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 20:39 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has joined #rust-bitcoin 22:29 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] --- Log closed Wed May 15 00:00:13 2019