--- Log opened Mon Jul 06 00:00:11 2020
10:29 -!- aaronc2347 [uid451246@gateway/web/irccloud.com/x-rhnklawfgzbcjcws] has joined #utreexo
10:44 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection]
10:56 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #utreexo
11:16 < kcalvinalvin> It's probably since I've only synced til 2014 but btcd is taking 82% of the time on gc and db (~40% on gc and ~40% on db)
11:16 < kcalvinalvin> Seems like utreexo thrown in there would help *a lot*
11:21 < ja> kcalvinalvin: how do you measure time spent in gc?
11:24 < rjected> kcalvinalvin: it's not because you've only synced till 2014, gc is crazy in btcd
11:25 < rjected> well maybe it is, depending on what you were expecting
11:25 < rjected> utreexo would help a ton
13:36 -!- ThomasV [~thomasv@unaffiliated/thomasv] has joined #utreexo
14:13 -!- ThomasV_ [~thomasv@unaffiliated/thomasv] has joined #utreexo
14:15 -!- ThomasV [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 264 seconds]
14:23 -!- ThomasV [~thomasv@unaffiliated/thomasv] has joined #utreexo
14:25 -!- ThomasV_ [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 258 seconds]
14:42 < adiabat> well... that seems like the next step then, shove utreexo into btcd
14:42 < adiabat> I actually don't think it'd be *that* hard
14:42 < adiabat> I mean fairly hard but not crazy
14:43 < adiabat> also now I have a server
14:46 < adiabat> there was a bunch of google cloud credits.  I spent like 2 hours just trying to figure out how to get ssh keys to work
14:46 < adiabat> anyway it's 35.188.186.244
14:47 < adiabat> I could give it a hostname I guess. bridgenode.tadge.net? or .lit3.co?  I have some random domains
14:49 < adiabat> I will try to have a bridge node of current master up and running on there, most of the time
14:49 < adiabat> also I guess if anyone wants I can put their ssh keys on it.  Or if you have your own servers you can do the same 
14:50 < adiabat> and we can have a binary randomly choose from a list
14:50 < adiabat> also I'm writing up some road map stuff and... I think we should have a quick "release" soon / now ish
14:50 < adiabat> just to get some interest
14:50 < adiabat> but then it's a bit weird because it seems like right after the release we switch to getting utreexo into btcd
14:51 < adiabat> which doesn't actually throw away too much code, but 'utreexoserver' and 'utreexoclient' would likely not make it
15:01 -!- instagibbs [~instagibb@pool-71-178-191-230.washdc.fios.verizon.net] has quit [Ping timeout: 256 seconds]
15:33 -!- instagibbs [~instagibb@pool-71-178-191-230.washdc.fios.verizon.net] has joined #utreexo
16:05 -!- ThomasV_ [~thomasv@unaffiliated/thomasv] has joined #utreexo
16:06 -!- ThomasV [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 246 seconds]
16:23 < adiabat> also, getting a "panic: varint is 7" at the end of genproofs
16:26 < adiabat> maybe should do this on github... but yeah why does rev.go #247 have that error?
16:26 < adiabat> if varint != 0, return error
16:26 < adiabat> then the next line is ti.Varint = varint... which must always be 0 at that point
16:27 < adiabat> so why even have that field if it's always 0?
16:27 < adiabat> anyway I'm going to take out the !=0 check there and see if everything still works fine
16:29 < adiabat> also ti.Height is always non-zero right?
16:29 < adiabat> since you can't spend anything from height 0
16:49 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection]
17:15 -!- dergoegge [~dergoegge@152.89.163.100] has joined #utreexo
18:04 -!- dergoegge [~dergoegge@152.89.163.100] has quit [Quit: leaving]
19:07 -!- pigeons_ is now known as pigeons
19:55 < kcalvinalvin> adiabat ti.Height is non-zero for all transactions *after* bitcoind 0.15. Before, they would all be 0 except for the last output
20:00 < kcalvinalvin> I think it's ok to get rid of the ```if varint != 0``` but ti.Height check is useful since wire.ReadVarInt is expensive
20:03 < kcalvinalvin> ja I've been using pprof, a golang profiling tool recently
20:06 < adiabat> kcalvinalvin: in the code it was if varint != 0 return error, which goes directly to a panic
20:06 < adiabat> so any non-0 varint would crash genproofs
20:07 < adiabat> anyway it's never used anywhere once it's read so I just took out reading it
20:07 < adiabat> we can put it back if it does something (other than panic)
20:07 < kcalvinalvin> I.. think you'd have to read it since I think it'll screw with reading the data that's after that
20:08 < adiabat> it is annoying that we have to read it to advance the pointer in the reader
20:08 < adiabat> yeah
20:08 < adiabat> it still crashes though at the end of genproofs for me
20:08 < kcalvinalvin> yeah I think crashing genproofs might have been a mistake for varint != 0 when height is >0. But odd how that didn't crash anything? Usually when varint was != 0, it meant that rev files were being read wrong
20:08 < adiabat> I dunno I have rev files that do that
20:09 < adiabat> and so did aaronc2347 I think
20:09 < kcalvinalvin> Yeah idk what's crashing it... 
20:09 < adiabat> I mean, rev files, and blk files I guess, are kind of mysteries
20:09 < adiabat> so we should try to recover or just say "OK we're done" if there's something weird in them
20:09 < kcalvinalvin> Apparently there are a handful of blocks that don't have a matching rev block
20:09 < adiabat> yeah I have that now
20:10 < adiabat> without the varint thing, I get an unexpected EOF panic
20:10 < kcalvinalvin> I guess that happens when you finish syncing and get bad blocks/not best blocks
20:11 < adiabat> I guess we should try to not panic on external data that's messed up, but try to ignore / recover maybe
20:11 < kcalvinalvin> I think you do need the varint since you don't know when the next data starts
20:11 < adiabat> yeah I'm still reading it, just ignoring it
20:11 < kcalvinalvin> ah ok
20:11 < adiabat> _, err := wire.ReadVarInt(r, pver)
20:12 < adiabat> except... it's always 1 byte, right?
20:12 < adiabat> are there any 2+ byte varints there for version?
20:12 < adiabat> it'd be nonstanard I guess but that doesn't mean they don't exist
20:12 < kcalvinalvin> I'm assuming it's not always 1 byte. But idk for sure
20:13 < adiabat> hmm.  I guess in compress.go there's tons of panics, but really they should be errors
20:14 < kcalvinalvin> I think for btcd they have all these checks before sending off to the functions in compress
20:15 < adiabat> in this case the last io.ReadFull fails
20:15 < adiabat> anyway we need to be able to reliably go to serving blocks at the end of genproofs, so ignoring errors here is OK
20:16 < adiabat> like if there's any error in rev blocks we just say "OK we're done!" and finish that loop and go to serving
20:17 < kcalvinalvin> lots to talk about today I guess heh
20:17 < adiabat> which is a bit of a hack but rev blocks are weird and we don't really know why they're out of order or not there, etc
20:17 < kcalvinalvin> Or tomorrow for you guys
20:17 < adiabat> yeah well that part is just... no panics 
20:17 < adiabat> I can do it now
20:17 < adiabat> or you can do it if you want as you're more familiar with the rev stuff
20:18 < kcalvinalvin> ignore errors?
20:18 < adiabat> no, return them
20:18 < kcalvinalvin> ah yeah maybe I can do them today
20:18 < adiabat> like func decompressScript(compressedPkScript io.Reader) ([]byte, error)
20:19 < adiabat> then I can catch an error in BlockAndRevReader and just say OK we're done
20:19 < adiabat> although... well I don't know if that will actually *work*
20:19 < kcalvinalvin> did you get a chance to look at the buffered offsetfile PR?
20:20 < adiabat> maybe it's something even weirder, and we really need to understand what the rev files are doing ...
20:20 < adiabat> but it seems like we pretty much have them OK, it's just the very end
20:20 < adiabat> yeah, looks good
20:20 < adiabat> maybe can merge tomorrow
20:21 < adiabat> oh somewhat related:
20:22 < adiabat> https://github.com/mit-dci/utreexo/commit/ed70f17fb83ad3a833afef0b600c4e8bb82f2b4a#diff-b0d0afbe26963186b20137292fb23458R581
20:22 < adiabat> threw me off for like an hour
20:23 < adiabat> the description of the read function:
20:23 < adiabat> Read reads up to len(b) bytes from the File.
20:23 < adiabat> It returns the number of bytes read and any error encountered.
20:23 < adiabat> apparently there's a semi-secret 1GB read limit...
20:24 < adiabat> ok I should go to bed for the call
20:24 < ja> late call tomorrow right?
20:24 < adiabat> yes, 12:30
20:24 < ja> all right!
20:24 < kcalvinalvin> 12:30 edt right?
20:24 < adiabat> yeah I guess switch back and forth
20:25 < adiabat> I could start at 12 if that's any better for KST
20:26 < kcalvinalvin> That'd be helpful
20:26 < ja> all right! 12!
20:27 < adiabat> ok 12 works
20:27 < kcalvinalvin> cool]
20:47 < aaronc2347> cool ill be there
21:46 -!- ThomasV_ [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 272 seconds]
--- Log closed Tue Jul 07 00:00:13 2020