--- Log opened Tue Jul 21 00:00:26 2020
00:20 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection]
00:24 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #utreexo
04:22 -!- aaronc2347 [uid451246@gateway/web/irccloud.com/x-bpmwofjwmlinmloy] has quit [Quit: Connection closed for inactivity]
06:02 < adiabat> ok utreexo call at 12 EDT but can do a quick call at 9:30 EDT for those in regions that can see the comet now
06:02 < fanquake> 👀
06:02 < adiabat> hm wait is the comet visible from southern hemisphere?  maybe not
10:13 -!- dergoegge [uid453889@gateway/web/irccloud.com/x-tvyewdtkwvbpaffm] has joined #utreexo
10:15 < dergoegge> forgot to mention this in the meeting but csn does not print the satoshi amount found at the end. #163 adds that
10:18 < dergoegge> adiabat: also would be nice if the csn remembers the utxos it found if it pauses and resumes, i will do that so you can concentrate on the backwards validation
11:25 < kcalvinalvin> adiabat I think we're reading one-by-one on network again
11:25 < kcalvinalvin> csn is taking 50% of the time on network
11:30 < kcalvinalvin> Ah yeah commit 2f2b8593c786de335115dbe780bb127f2a2069e6 undoes some of commit fd7287d17e89cb8dccae210ce99db22fa5c1c97f
11:32 < kcalvinalvin> When we call deserialize(con) every io read is through the wire. So it does a lot of small reads through the wire tons of times
12:03 < adiabat> dergoegge: yeah a persistent wallet could be cool, just need to serialize outpoints and amounts, so it's all fixed length
12:04 < adiabat> no pubkeys is nice because those are variable length
12:04 < adiabat> kcalvinalvin: ah OK so I made the server faster but the client slower.  oops
12:05 < adiabat> can change that back in #164 which I can merge soon
12:32 < kcalvinalvin> adiabat I have a fix coming
12:32 < kcalvinalvin> Unless you're super close of course :)
12:48 < adiabat> oh, yeah I am heh
12:49 < adiabat> using io.CopyN(&buf, con, int64(ublen))
13:00 < kcalvinalvin> oh
13:00 < kcalvinalvin> Well I'm glad I made them two separate commits now
13:01 < kcalvinalvin> adiabat I made a PR
13:08 < adiabat> which does this fix
13:08 < adiabat> the ublock deserialize is annoying... why is it slower to deserialize from the tcp connection?
13:08 < adiabat> if that's the case we shouldn't have a func (ub *UBlock) Deserialize() function, I'm making a func (ub *UBlock) FromBytes(argbytes []byte)
13:09 < adiabat> not sure why it'd be faster though
13:10 < kcalvinalvin> When the connection is passed as an io.Reader to deserialize, it reads from wire whenever you read from it
13:11 < adiabat> right but is that slower?  The data from the network goes into a ram buffer either way
13:11 < kcalvinalvin> It doesn't go in the buffer though
13:11 < kcalvinalvin> It literally just reads 4 bytes at a time from the tcp connection
13:12 < kcalvinalvin> like this bit `err = binary.Read(r, binary.BigEndian, &uDataLen)` is read directly from the wire without buffers
13:14 < adiabat> anyway OK I have it with ub.FromBytes now, it works
13:14 < adiabat> hmm conflicts with your PR though
13:15 < adiabat> but I think only in the utils.go UblockNetworkReader part?
13:15 < adiabat> your PR is bigger so how about I merge mine first?
13:16 < kcalvinalvin> Oh wait do you mean like the tcp itself buffering?
13:16 < kcalvinalvin> Yeah sure you can merge first
13:16 < adiabat> yeah I assume the OS reads it into a buffer
13:16 < adiabat> if it's faster this way though cool
13:17 < kcalvinalvin> I'm guessing it's just efficiencies coming from reading a big chunk at once?
13:18 < adiabat> ok should I merge mine then you change yours?  I think the conflicts are mostly comments and stuff
13:26 < kcalvinalvin> adiabat sure. ja did have a review on a comment (which I did address) but haven't heard back. It is a comment so can improve on it later though
13:27 < adiabat> the return // give up and don't make hashable node part?
13:27 < adiabat> that might be an error
13:27 < kcalvinalvin> yeah
13:27 < adiabat> I'm not sure that ever happens
13:27 < adiabat> it feels like there should be a hn=nil there
13:28 < kcalvinalvin> Is there a case where both children could be nil during rem2
13:28 < adiabat> well it doesn't do anything so the comment is wrong
13:29 < adiabat> the whole if nsib == nil || n == nil {} is useless
13:29 < adiabat> since it just returns anyway
13:33 < kcalvinalvin> adiabat removed that bit
13:34 < adiabat> ok
13:34 < adiabat> the integration tests download bitcoind every time
13:34 < adiabat> and it takes most of the time
13:34 < adiabat> hmm maybe we can cache it or something...?
13:37 < adiabat> I don't know why the integration tests are failing
13:38 < adiabat> it just says ##[error]The operation was canceled.
13:38 < adiabat> but... that's the only error
13:39 < kcalvinalvin> Is this is free tier 5 min thingy
13:39 < adiabat> maybe]
13:40 < adiabat> it's also probably something with the knownTipHeight thing
13:40 < adiabat> there's no other way to test though, can't rebuild each time so have to do it shorter
13:41 < kcalvinalvin> Oh you have multiple commits so I think github might be treating that whole thing as one github actions run
13:41 < kcalvinalvin> or not... idk how actions work
13:43 < adiabat> yeah it spends the majority of the time wgetting the bitcoin tar file
13:43 < adiabat> maybe we can just make a repo on github with that tar file then it will go faster?
13:44 < ja> that would work, but is ugly since binaries are never pruned unless you force push
13:44 < adiabat> yeah 2m25s to download at 200KB/sec, 2m 38s total time
13:44 < ja> i will try to set up some nix based CI like i wanted to earlier...
13:44 < adiabat> maybe a separate repo
13:44 < adiabat> as long as it's somewhere on github, maybe it will download faster
13:44 < adiabat> or gist or something
13:45 < adiabat> just so it's in their data center somewhere
13:45 < ja> the correct way using github actions would probably be to provide an setup-* environment, like the way we get go
13:45 < ja> those surely must be cached
13:45 < adiabat> anyway whatever I'm merging it
13:45 < adiabat> tests still running but whatever
13:46 < ja> the issue with doing it properly with actions is that it is a closed ecosystem
13:47 < ja> and if i remember correctly, we are only using actions because they provide free CI time and it is integrated, not because it is the best
13:47 < adiabat> yeah
13:47 < adiabat> one side or the other is limiting download speed because it's always right at 200K/sec
13:48 < adiabat> can try putting the binary somewhere on github and see if that speeds it up
13:53 < adiabat> hm.  I dunno, it still says 
13:53 < adiabat> 2020-07-21 20:49:21 running idbsim... ##[error]The operation was canceled.
13:54 < adiabat> kcalvinalvin: I can change the conflicts in #165 or you can
13:54 < kcalvinalvin> adiabat already ahead of you
13:55 < kcalvinalvin> Pushed the merge :)
13:55 < adiabat> ah ok
13:55 < kcalvinalvin> oh oops
13:55 < kcalvinalvin> wait I didn't commit the change
13:56 < dergoegge> sorry guys i went for dinner and took longer than expected. i will have a look at the integration tests. looks like ibdsim never exits and than it runs for 5+ minutes and gets canceled.
13:59 < adiabat> it's OK, I'm pretty sure it's still working fine
14:00 < adiabat> not a big deal, but it at some point we should try to get the tar file closer to where it's running
14:04 < dergoegge> i have that 200k limit on my mac as well i think bitcoin.org does some ratelimiting
14:08 < dergoegge> i know whats going on, the csn does not quit if it got to the last height. you removed the EOF check and channel close thing i put in there.
14:10 < adiabat> ahh shoot
14:10 < adiabat> oops
14:12 < kcalvinalvin> Well I'll be pushing changes so I think it's fine?
14:15 < adiabat> I have a 2 line change to master to close the channel and let ibd finish
14:15 < adiabat> can do it after merging yours or before
14:18 < adiabat> ok pushed that fix to master
14:19 < adiabat> wait no I didn't
14:25 < adiabat> ok now works
14:26 < adiabat> kcalvinalvin: is #165 OK to merge?
14:26 < dergoegge> could not really test #166 because i need to reIBD
14:26 < kcalvinalvin> adiabat no everything broke and I lost the commit so I have to rewrite the code
14:27 < adiabat> yeah I have the commented out // knownTipHeight = 260500
14:27 < adiabat> that lets you genproof and IBD fast
14:27 < adiabat> kcalvinalvin: aw that's annoying
14:27 < adiabat> good ol git :)
14:28 < kcalvinalvin> Could we do 1 commit per feature from going forward? These multicommits really bog down rebasing
14:28 < adiabat> squash everything?
14:28 < adiabat> yeah we can do the squash merge thing
14:29 < dergoegge> hm but do you have an address for me with utxos before height 260000? 
14:30 < adiabat> oh shoot no
14:30 < adiabat> because it only does bech32...
14:30 < kcalvinalvin> I made a new PR with only the pollard changes
14:34 < dergoegge> adiabat: do you have a pollardFile at height ~1500000? then i could tweak it to start from there. 
14:34 < adiabat> I do!
14:34 < adiabat> can almost post it on IRC
14:34 < dergoegge> utreexo being useful already!
14:34 < adiabat> ok this is dumb but lets do it on IRC
14:35 < adiabat> here's height 1443111, base64 encoded
14:35 < adiabat> ABYFJwAAAAABN6MIbmBJ/IomZ2f7uxn4jEUDTJgyMvjUTL4RjnMou0kjD/sSiwrk87AQf/O5gTm2
14:35 < adiabat> 7kMKtK0hKs3m61OKaSa68eCMYyp3yR4f6lh6HbguPMaLkyZDWDvjCZfvs4HY6FCkAzzA4Ewj+JYN
14:35 < adiabat> PNrB4HpXCV51kAbgbhQtwI3Hl1uR6ENM987TzoPeW/au4ayHz2vMemNFQ0mdP7k4adzguZctkSAo
14:35 < adiabat> r0WPgu1XrR59h5NIWGW313XViQsMSNMuLzGJWmfWXIFJMpJga7nO4CMVJ7J9PxiRIK2F7Say4+xd
14:35 < adiabat> t4AygLvConA9N0QCYqZ76icE+ctNZ7jHtNK+TR2od86+vO2mQ7/nSnAWjrcPencaGLl6Uwc68lr9
14:35 < adiabat> Q/PGtVKKazvNKWlILqVdO5MyhinQ12PZB4Ul1HZcplxlhAO3DSE7HU9bNIcgN3HJ6AwNyzxF60lw
14:35 < adiabat> 0H+iKVotJCfUzRB/qnxtBBZ9kwGYzozNcTNfBKEODtPoy+PRoxFqJaNNG0BEAKznGejAOrkmzMzb
14:35 < adiabat> 7kZARKE501397VdTVNxD54GGGI1JQTMzI7SLij1bSNAsvgUUaZ+6ST9YogV9EljyYGc8ILkr4cma
14:35 < adiabat> yhwA8KyISKIZKYhYhvFwsKp8FAeYFBuHefp6MtuwIUapgwCW2/aMhLlu4V8nd37HeGNqr9qna1E8
14:35 < adiabat> bDTEzJp18gR1R4r95/C2XxRHMav9HzVrj72F9fMM5KwlQE3BO911/xpkbA==
14:35 < dergoegge> haha ok :D the height is already in there right?
14:35 < adiabat> yup it should know on its own
14:36 < adiabat> the start is 0016 0527 0000 0000 0137 a308
14:36 < adiabat> 0016 0527 is height 1443111, then 0000 0000 0137 a308 is 20423432 leaves
14:37 < adiabat> 20423432 has a bunch of roots; maybe have a function that looks for "small" root sets, where there are only 3 or 4 roots
14:38 < adiabat> then encoded it would only be like 4 lines
14:46 < dergoegge> nice now i just need to sync to block 1719999. if you have a even closer pollardFile to that height let me know. but it should not take long from here i think
14:50 -!- rjected[m] [rjectedmat@gateway/shell/matrix.org/x-kedvajnlvydrfmtf] has joined #utreexo
14:51 < dergoegge> btw. with #166 the pollardFile holds more stuff: number of utxos found + the utxos(-PKScripts)
14:52 < dergoegge> so if you need your old pollardFiles to work you can prepend 4 zero bytes and it will work
14:55 < adiabat> ah ok
14:55 < adiabat> I mean if this were a real thing probaly wallet and pollard should be separate but whatever :)
14:55 < adiabat> wow it's so much slower when not connecting to localhost
14:56 < adiabat> but that makes sense as my internet is slow
14:57 < adiabat> but wonder if there's latency problems, like it's not queuing properly
14:57 < adiabat> I can merge it as it works, but maybe the network code is slow
15:01 < adiabat> hmmm yeah latency is a problem, especially with small blocks in the beginning, since it's requesting 1 block at a time
15:02 < adiabat> worth fixing now...?  could request a bunch of blocks at once.. hm
15:03 < dergoegge> is it a lot slower than before?
15:03 < adiabat> not for localhost
15:03 < adiabat> there was no connecting to remote servers before
15:04 < adiabat> so localhost is still fine, but now that it can actually connect over the internet, I'm realizing for small blocks it's slow
15:04 < adiabat> once the blocks get to large size it's doesn't matter, but with testnet there are lots of empty blocks
15:04 < ja> is it making a new connection for every block?
15:05 < adiabat> no, same connection
15:05 < adiabat> just it doesn't request block n+1 until it gets block n
15:05 < adiabat> so max blocks / second is your ping time
15:05 < adiabat> uh 1/ping time 
15:07 < adiabat> hmm... I might try backwards then fix the network latency
15:07 < ja> would it make sense to request ranges of blocks?
15:07 < adiabat> ja: yeah that would be better
15:07 < adiabat> like the client sends 2 bytes for number of blocks, then 4 bytes each for the block height requested
15:08 < ja> i'm not gonna do it, thought, i am working on the CI now
15:08 < adiabat> then the server grabs em all and sends it onto the network in order
15:08 < adiabat> not hard to fix but maybe do it tomorrow after "release"
15:08 < adiabat> then v0.2 can be much faster than v0.1 heh
15:24 < dergoegge> i think there was still a small bug in #165. i should have a fix in 5min
15:28 < dergoegge> #169 is the fix. i tested it should work now
15:29 < adiabat> ah ok
15:31 < dergoegge> ops the bug was in my pr #166 not #165
15:45 < adiabat> wait 169 is OK though right?
15:46 < dergoegge> yes
15:50 < dergoegge> are you working on requesting ranges of blocks? otherwise i could look into it
15:51 < adiabat> I'm working on backwards now
15:52 < dergoegge> OK i will look into ranges
15:52 < adiabat> which touches some of the same code, so may be a little tricky to merge but should be doable
15:52 < adiabat> ok
15:54 < adiabat> I have the backwards PR open if looking at that helps
15:56 < dergoegge> ok i will rebase once you are done, should not create a complex conflict
16:44 < dergoegge> i took a slightly different approach with the ranges than you outlined above. the csn sends two int32s, height a and height b. if a < b the bridge sends blocks a+1, a+2, ... b if b < a the bridge sends blocks a, a-1, a-2, .... b.
16:46 < adiabat> ah OK that makes sense
19:00 < adiabat> hmmm I don't know if I can do backwards tonight
19:00 < adiabat> there's a bunch of things that need to change
--- Log closed Wed Jul 22 00:00:27 2020