--- Log opened Sun Apr 28 00:00:58 2019 00:36 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 00:40 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 258 seconds] 01:04 -!- ccdle12 [~ccdle12@p116127-ipngn200301takamatu.kagawa.ocn.ne.jp] has quit [Remote host closed the connection] 01:07 -!- TamasBlummer1 [~Thunderbi@p200300DD670F9D156506B7C4D522E0CB.dip0.t-ipconnect.de] has joined #rust-bitcoin 01:09 -!- TamasBlummer [~Thunderbi@p200300DD670F9D506506B7C4D522E0CB.dip0.t-ipconnect.de] has quit [Ping timeout: 264 seconds] 01:09 -!- TamasBlummer1 is now known as TamasBlummer 02:32 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-wyudchrkjgptfkxx] has joined #rust-bitcoin 02:36 -!- ccdle12 [~ccdle12@p116127-ipngn200301takamatu.kagawa.ocn.ne.jp] has joined #rust-bitcoin 04:10 -!- dmkathayat [~quassel@ec2-13-233-97-253.ap-south-1.compute.amazonaws.com] has joined #rust-bitcoin 05:40 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-wyudchrkjgptfkxx] has quit [] 05:40 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-lmtklgocxouyqwhn] has joined #rust-bitcoin 07:00 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection] 07:00 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has joined #rust-bitcoin 07:09 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection] 08:09 -!- ccdle12 [~ccdle12@p116127-ipngn200301takamatu.kagawa.ocn.ne.jp] has quit [Remote host closed the connection] 08:10 < stevenroose> sgeisler: looking into that on Monday, currently experimenting with something for @shesek 08:11 < stevenroose> andytoshi, sgeisler: I'd like to use the bitcoin_hashes macros outside of bitcoin_hashes.. The modules are private. Would it be ok to (1) make them public, or (2) add an "internals" non-default feature that exposes them? 08:12 < sgeisler> stevenroose: which macros are you talking about specifically? 08:13 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-lmtklgocxouyqwhn] has quit [Quit: Connection closed for inactivity] 08:18 < stevenroose> all of them, kinda what you need to implement your own hash 08:19 < stevenroose> hex_fmt_impl, index_impl, borrow_slice_impl, serde_impl and perhaps the write_impl for Engine. 08:19 < stevenroose> I'm trying to create an implementation of Hash for the special hash we use in Elements. 08:19 < stevenroose> It's kind of a sha256 midstate. 08:20 < stevenroose> I'd like to give it a name and have a way to compute them. It's just a bit awkward because it uses the "sha256 engine" but also not, because the end result is a midstate. So to specify an Engine I don't know if I can specify the sha256 engine. 08:22 < stevenroose> the sha256d hash also specifies engine: sha256::Engine 08:23 < stevenroose> But then in from_engine etc, it can just finalize the sha256 and add hash another round. With midstate is kind of the exact opposite where you should have an engine that does less. I'm also not super familiar with the concept of midstate in SHA, so I'm just looking into it and figuring it out :) 08:23 < stevenroose> The nice thing with this kind of implementations is that you have very easy testability. If you have any mistake, it just won't work. 08:28 < sgeisler> implementing your own hash types sounds like a reasonable use case 08:29 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 08:29 < sgeisler> are you sure the existing midstate API isn't sufficient for your immediate use case? 08:42 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 08:50 < andytoshi> stevenroose: if the midstate API doesn't work for rust-elements please PR to fix the midstate API 08:50 < andytoshi> that API literally only exists for elements 08:51 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 08:59 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 08:59 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 09:01 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 09:01 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 09:05 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 09:05 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 09:06 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 09:06 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 09:10 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 09:10 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has joined #rust-bitcoin 09:13 < stevenroose> andytoshi: the midstate API works in the sense of being able to calculate it 09:15 < stevenroose> but I don't think the midstate type is useful. It's not ergonomic f.e. to have a field in `pub witness_hash: sha256::Hash::Midstate` because that's just a type alias to `[u8; 32]` and doesn't have Display, FromHex or serde::Serialize etc implemented. 09:16 < stevenroose> So I was thinking of a `pub struct elementshash::Hash(u8; 32])` that has a trivial `bitcoin_hash::Hash` implementation that just uses the midstate function from `sha256::HashEngine` but that we can then use as a proper newtype with traits defined on it. 09:20 < stevenroose> F.e. AssetIds are currently implemented using sha256::Hash in the liquid-rpc crate, but should be of this other hash type because they are not proper SHA256 hashes. 09:26 < stevenroose> Personally I could do with just a simpler `pub struct AssetId([u8; 32])` newtype and ignore the incnsistency for the witness hash. 09:26 < stevenroose> We don't expose that witness hash anywhere else than liquid-rpc crate. 09:26 < stevenroose> And then I can just implement the traits without the macros. 09:27 < stevenroose> I hate it when I [patch] a dependency (f.e. bitcoin_hashes), then the types become different then unpatched types from dependent crates. So elements's bitcoin_hashes != bitcoin's bitcoin_hashes and the BitcoinHash trait impl suddenly is wrong because of mismatched types :/ 09:55 < stevenroose> What would be the Rust equivalent for `while (!(count & (((uint32_t)1) << level)))`? The count and level variables are u32 defined, so I have `while count & (1 << level) == 0` 09:55 < stevenroose> I can make it 1u32 though. 10:00 < andytoshi> stevenroose: ok, I think we should give `sha256::Hash::Midstate` its own type 10:00 < andytoshi> and use all the macros to make it useful 10:01 < andytoshi> and expose this in bitcoin_hashes as sha256::Midstate 10:02 < stevenroose> That's also an idea. 10:03 < stevenroose> But like have sha256::Midstate also implement Hash f.e.? 10:03 < stevenroose> That's curious :D 10:03 < stevenroose> But I guess it would work. I can do that. I'm going out for a bit now, but tonight I might. I actually worked around it for now, but I think that idea is better. 10:05 < andytoshi> i don't think it should implement Hah 10:05 < andytoshi> Hash 10:05 < andytoshi> it shouldn't need to 10:05 < andytoshi> but FromHex and Serialize/Deserialize sure 10:05 < andytoshi> and Borrow<[u8]> 10:05 < andytoshi> I agree this will make it annoying to compute with. it should be. it's really dangerous :} 10:39 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has quit [Ping timeout: 256 seconds] 11:11 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 11:15 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 250 seconds] 12:08 < stevenroose> andytoshi: Hmm, the Hash trait has most of the useful methods though. from_slice, into_inner, from_engine... 12:09 < stevenroose> Meh, I can implement those traitless. 12:28 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has joined #rust-bitcoin 12:37 -!- BlueMatt [~BlueMatt@unaffiliated/bluematt] has joined #rust-bitcoin 12:47 < andytoshi> well, we don't want `hash()` or `from_engine()` 13:20 -!- Dean_Guss [~dean@gateway/tor-sasl/deanguss] has quit [Ping timeout: 256 seconds] 13:24 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 13:30 < stevenroose> I added `from_sha256_engine`. I think semantically it's correct, it's the midstate so you can get it from the engine by calling `.midstate()`. I'm having trouble replicating the `CalcFastMerkleRoot` method. Let me push. 13:37 < stevenroose> https://github.com/rust-bitcoin/bitcoin_hashes/pull/39 13:42 < stevenroose> https://github.com/ElementsProject/rust-elements/pull/19 13:56 < stevenroose> andytoshi: I have the same feeling like with blech32 last week: staring blank at a C++ and a Rust implementation and somehow it's not working out. 13:58 < andytoshi> why do you need a from_sha256_engine function? i really don't like the idea of people treating the midstates as a fundamental object 13:58 < andytoshi> vs a side-effect of the sha256 engine 14:00 < andytoshi> also btw there's a 50% chance that the hex encoding is backward, since we're defining this object based on a Core-ism 14:22 -!- instagibbs [~instagibb@pool-100-15-135-248.washdc.fios.verizon.net] has quit [Ping timeout: 246 seconds] 15:08 < stevenroose> yeah I asked Greg about backwardness :D I honestly can't easily find out in Core if it is reversed. Internally it's a uint256. 15:09 < andytoshi> if it's internally a uint256 it's backwards 15:09 < stevenroose> So it probably is, since block hashes are also uint256. 15:09 < andytoshi> one sec, i'm going to find some liquid issuances we can use for unit tests 15:09 < andytoshi> yep 15:09 < andytoshi> so if you're lucky, that's the only reason your tests are failing 15:10 < stevenroose> Nah I tried. In fact the test strongly suggests it's not backwards. 15:11 < andytoshi> which test? 15:11 < stevenroose> I put it on backwards now and then the 2nd vector fails instead of the 3rd (it's a sequence of n-sized trees) and it expects the reverse hash. 15:11 < stevenroose> The bottom one in merkle_test.cpp in elements. 15:11 < andytoshi> ah cool 15:12 < stevenroose> It's just a vector of leaves and vector of expected roots that Greg made a few months back because we were also adding support for that hash to the python tools for testing. 15:12 < stevenroose> (Guess what? We couldn't find a midstate python lib so just have a dev-only RPC for the merkle tree function :|) 15:12 < andytoshi> :/ 15:13 < stevenroose> ah wait, I decode the hex as Vec and encode as Midstate, that's probably a bug 15:14 < stevenroose> because one does not reverse 15:14 < andytoshi> heh yes. in general you should not decode as Vec ever 15:14 < andytoshi> just decode directly to/from the type you care about 15:15 < andytoshi> then the reversals should be consistent (unless your de/encoding code is wrong :)) 15:15 < stevenroose> ah nice, tests finish! 15:15 < andytoshi> also Vec allocates heap memory which will kill your performance 15:15 < stevenroose> ah, about that 15:16 < andytoshi> copying 32 bytes around should be less than a nanosecond. malloc could easily be microseconds. 15:16 < stevenroose> I notice that the generic FromHex impl for Hash first allocated into a vec and then allocated the Hash type with from_slice 15:16 < stevenroose> I think it's possible to not have that double allocation 15:17 < andytoshi> ouch really? 15:17 < andytoshi> did i do that? 15:17 < stevenroose> Don't know who. It's because you can't FromHex the array directly because you don't know the size, it's Self::LEN. It would work if you have a macro instead of a generic trait impl. 15:18 < stevenroose> I think I can PR that, though. 15:18 < andytoshi> looks like it was me 15:18 < andytoshi> cool 15:18 < andytoshi> yeah, that's really bad. i wonder what i was thinking 15:22 < stevenroose> Ah is it possible to do something like this `let inner: Self::Inner = FromHex::from_hex(&s)?;` where the Inner is the type on Hash? 15:23 < stevenroose> I think somehow you'd need ::Inner or something. 15:24 < stevenroose> Ah no, you can't do the newtype cast in that case. 15:24 < stevenroose> k, nvm, needs macro, on it 15:24 < andytoshi> are you sure 15:24 < andytoshi> i think that will work 15:24 < andytoshi> you need to have the `Hash` trait require `Inner: FromHex1` 15:24 < andytoshi> you need to have the `Hash` trait require `Inner: FromHex` 15:24 < andytoshi> which is fine 15:25 < andytoshi> you don't do a "newtype cast", you can use the existing from_inner() method on the trait 15:26 < andytoshi> like, can't you write `Self::from_inner(Self::Inner::from_hex(&s)?)` ? 15:26 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 15:27 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 15:29 < stevenroose> ah, didn't know about from_inner 15:29 < stevenroose> Yeah that would work... 15:31 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 245 seconds] 15:34 < andytoshi> while you're at it can you add a `to_inner` :) 15:35 < andytoshi> oh, wait, maybe we have to_inner and it's from_inner that doesn't exist.. 15:35 < andytoshi> anyway i just needed `from_inner` earlier today so pls add it if it's not there :} 15:38 < stevenroose> yeah did that 15:38 < stevenroose> into_inner exists 15:38 < andytoshi> ah 15:38 < andytoshi> yeah, `from_inner` seems like a really big oversight 15:39 < stevenroose> For your trick to work, I had to add `type Inner: FromHex` as an extra constraint.. 15:39 < andytoshi> yep 15:40 < andytoshi> I think that's perfectly reasonable 15:41 < stevenroose> andytoshi: do you prefer two separate PRs? One for the hex and inner fixes and one for Midstate? 15:41 < stevenroose> I'll remove the Midstate::from_sha256_engine method then. 15:42 < stevenroose> It's basically a shorthand for Midstate::from_inner(engine.midstate()) though 15:42 < andytoshi> engine.midstate() should return a Midstate directly 15:42 < andytoshi> and sure, one PR is fine, i'll try to review it as soon as it's up 15:49 < stevenroose> https://github.com/rust-bitcoin/bitcoin_hashes/pull/40 15:50 < stevenroose> ah wow, I read your "yep" as being an answer to my question that actually came later. 15:50 < andytoshi> hmmm 15:50 < stevenroose> PRing midstate separately then 15:50 < andytoshi> no worries 15:50 < andytoshi> i wonder if we can get rid of the vec even in the reversed case 15:50 < stevenroose> I doubt it. :( unless we write a special case parser of course 15:51 < andytoshi> i mean, we could stick a `reverse` method on the FromHex trait :P 15:51 < andytoshi> then we could do T::Inner::from_hex(&s).reverse() 15:52 < andytoshi> this wolud be irritating to implementers, especially people who are only using non-insane hashes 15:53 < andytoshi> but also like, people who aren't doing bitcoin stuff have lots of hex crates to choose from.. 15:55 < stevenroose> I think a reverse parser method would be simpler, no? 15:55 < andytoshi> oh yeah 15:55 < andytoshi> maybe we have one actually? 15:55 < andytoshi> we have a reverse serializer.. 15:56 < andytoshi> can you add a reverse parser? 15:56 < stevenroose> In fact, it's only [u8; 32] that we need reversed, no? 15:56 < stevenroose> hmm, the reverse parser is actually hard to introduce I just realize 15:56 < stevenroose> since we all leverage the [u8; 32]::from_hex now 15:57 < andytoshi> i don't understand 15:57 < stevenroose> Also the fact that Midstate is serialized backwards and sha256::Hash is not is kind of an Elements thing that is unrelated to bitcoin. 15:58 < andytoshi> lol, it's related to bitcoin in that it's an incredibly satoshi situation to be in 15:58 < stevenroose> What I mean is that the actual parsers are implemented in the macro FromHex for [u8; $len] and in FromHex for Vec. 15:59 < andytoshi> we can add a `from_hex_reversed` method which does basically the same thing 15:59 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 15:59 < stevenroose> So since FromHex for T: Hash uses the FromHex defined on the Inner type 15:59 < andytoshi> and just reverses the [u8; $len] after 15:59 < stevenroose> hmm, ah yeah we could do that :D 15:59 < andytoshi> that's my vote 16:00 < andytoshi> (i can see why i did the Vec thing now..) 16:01 -!- ccdle12 [~ccdle12@p116127-ipngn200301takamatu.kagawa.ocn.ne.jp] has joined #rust-bitcoin 16:03 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 258 seconds] 16:04 < stevenroose> Uhm.. Now we get a funny situation :D 16:04 < stevenroose> I implement from_hex on Hash: I have to do `if Self::BACKWARDS { Inner::from_hex_backwards } else { Inner::from_hex }` (pseudocode) 16:05 < stevenroose> But then.. I have to implement from_hex_reversed on Hash :D 16:05 < stevenroose> So from_hex_reversed will be the non-reversed implementation for BACKWARDS Hashes and reversed otherwise :D 16:05 < stevenroose> madness 16:08 < stevenroose> Adding a reversed: bool to HexIterator, is it worth it to have this one allocation less for reversed hashes? 16:10 < andytoshi> haha 16:10 < andytoshi> hmmm 16:10 < andytoshi> no, this is getting crazy 16:10 < andytoshi> wait, what is HexIterator? 16:11 < stevenroose> some thing that is used internally to parse hex 16:11 < andytoshi> ah i see 16:12 < andytoshi> so, what i'd like is for HexIterator to internally use a byte iterator 16:13 < andytoshi> and for FromHex to work with a byte iterator actually 16:13 < andytoshi> and then the existing `from_hex` method will be a convenience method that calls the iterator-based method on `s.bytes()` 16:13 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 16:13 < andytoshi> and then in our generic from_hex code, it can alternately use `s.bytes().rev()` 16:14 < stevenroose> That won't work 16:15 < stevenroose> If you use the iterator .bytes() you do [0]<<4+[1] and with .bytes().rev() you do [1]<<4+[0]. 16:15 < andytoshi> oh right 16:15 < andytoshi> we'd have to write our own stupid adaptor 16:15 < stevenroose> Anyway, I'm going to bed, can look at this more tomorrow. For now we just allocate for reverse. Sadly we mostly use reversed. 16:15 < andytoshi> because the rustc developers rejected my proposal to add this because it was "too niche" 16:15 < stevenroose> I'll keep it at that and go to bed. 16:15 < andytoshi> then dropped IRC to avoid me 16:15 < stevenroose> :D 16:16 < andytoshi> ok, cool. i'll think about this 16:16 < stevenroose> Updated the PRs. 16:17 < andytoshi> i think we want to impl DoubleEndedIterator on HexIterator 16:17 < andytoshi> then we can `.rev()` the HexIterator itself 16:19 < andytoshi> then we change the FromHex trait to have an internal method that simply takes a byte-iterator 16:19 < andytoshi> and `from_hex` will pass it a HexIterator 16:19 < andytoshi> or a reversed HexIterater, as the case may be 16:22 < stevenroose> Addressed your comments on midstate PR 16:22 < stevenroose> gtg now 16:23 < andytoshi> kk have a good night! 17:16 -!- BlueMatt [~BlueMatt@unaffiliated/bluematt] has quit [Ping timeout: 258 seconds] 17:38 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 17:39 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 17:41 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 17:41 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 18:05 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 18:05 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 18:10 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 246 seconds] 19:51 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 20:19 -!- ccdle12 [~ccdle12@p116127-ipngn200301takamatu.kagawa.ocn.ne.jp] has quit [Remote host closed the connection] 20:58 -!- ccdle12 [~ccdle12@p116127-ipngn200301takamatu.kagawa.ocn.ne.jp] has joined #rust-bitcoin 22:43 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #rust-bitcoin 22:46 -!- willpiers [~willpiers@38.75.231.30] has quit [Remote host closed the connection] 22:46 -!- willpiers [~willpiers@38.75.231.30] has joined #rust-bitcoin 22:50 -!- willpiers [~willpiers@38.75.231.30] has quit [Ping timeout: 246 seconds] 23:02 -!- ccdle12 [~ccdle12@p116127-ipngn200301takamatu.kagawa.ocn.ne.jp] has quit [Remote host closed the connection] 23:02 -!- ccdle12 [~ccdle12@p116127-ipngn200301takamatu.kagawa.ocn.ne.jp] has joined #rust-bitcoin 23:38 -!- gnusha [~gnusha@unaffiliated/kanzure/bot/gnusha] has quit [Ping timeout: 250 seconds] 23:39 -!- gnusha [~gnusha@unaffiliated/kanzure/bot/gnusha] has joined #rust-bitcoin 23:39 -!- Topic for #rust-bitcoin: Order of the Rust-Bitcoin Libraries Maintainers. Artisinal Bitcoin for the rest of us. Do NOT have private conversations on the FreeNode network! | https://github.com/rust-bitcoin | Logs at http://gnusha.org/rust-bitcoin/ 23:39 -!- Topic set by BlueMatt [~BlueMatt@unaffiliated/bluematt] [Tue Jan 22 10:11:18 2019] 23:39 [Users #rust-bitcoin] 23:39 [ andytoshi ] [ dmkathayat ] [ gnusha ] [ midnightmagic ] [ sgeisler ] [ warren ] 23:39 [ ariard ] [ dongcarl ] [ gribble ] [ molz ] [ simian_za ] [ windsok] 23:39 [ belcher ] [ dpc ] [ gwillen ] [ moneyball ] [ slimcognito ] [ wraithm] 23:39 [ Blackwolfsa] [ e4xit ] [ harding ] [ neonknight646 ] [ stanimal ] [ wumpus ] 23:39 [ ccdle12 ] [ Ed0 ] [ jamesob ] [ nickler ] [ stevenroose ] 23:39 [ CjS77 ] [ esotericnonsense] [ kallewoof ] [ nothingmuch ] [ takinbo ] 23:39 [ CubicEarth ] [ ghost43 ] [ kanzure ] [ real_or_random] [ TamasBlummer] 23:39 [ DeanGuss ] [ gmaxwell ] [ michaelsdunn1_] [ schmidty ] [ valwal_ ] 23:39 -!- Irssi: #rust-bitcoin: Total of 44 nicks [0 ops, 0 halfops, 0 voices, 44 normal] 23:39 -!- Channel #rust-bitcoin created Fri Mar 9 09:46:56 2018 23:39 -!- Irssi: Join to #rust-bitcoin was synced in 54 secs --- Log closed Mon Apr 29 00:00:58 2019