--- Log opened Mon Jun 08 00:00:44 2020
02:01 -!- rafalcpp [~racalcppp@ip-178-214.ists.pl] has joined #utreexo
03:38 -!- ThomasV [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 272 seconds]
04:13 -!- ThomasV [~thomasv@unaffiliated/thomasv] has joined #utreexo
04:37 -!- ThomasV [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 272 seconds]
04:45 -!- instagibbs [~instagibb@pool-71-178-191-230.washdc.fios.verizon.net] has quit [Ping timeout: 256 seconds]
04:46 -!- instagibbs [~instagibb@pool-71-178-191-230.washdc.fios.verizon.net] has joined #utreexo
04:48 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has joined #utreexo
06:16 -!- ThomasV [~thomasv@unaffiliated/thomasv] has joined #utreexo
07:22 -!- rafalcpp [~racalcppp@ip-178-214.ists.pl] has quit [Excess Flood]
07:24 -!- rafalcpp [~racalcppp@ip-178-214.ists.pl] has joined #utreexo
07:43 -!- adiabat [~adiabat@63.209.32.102] has joined #utreexo
08:11 < kcalvinalvin> PR #144 is ready for review/merge :)
08:15 < adiabat> oops I rebooted this and was off IRC for a day ish
08:15 < adiabat> hopefully didn't miss anything too critical
08:16 < adiabat> neato, does #144 mess up core's levelDB? or safe to try on a synced up one
08:16 < kcalvinalvin> adiabat that problem is gone :)
08:21 < adiabat> cool.  will try it
08:21 < adiabat> does it work if bitcoind is still running?
08:21 < kcalvinalvin> Nope. Leveldb doesn't support 2 processes accessing a single db
08:21 < adiabat> I mean you can read files when someone else has a lock on em right..?
08:21 < adiabat> hmm
08:22 < kcalvinalvin> rocksdb (fork of leveldb) does though
08:22 < kcalvinalvin> heh
08:22 < adiabat> so you have to shutdown bitcoind to run genproofs...
08:22 < kcalvinalvin> yes
08:22 < adiabat> & obviously there's no RPC to tell you anything about where rev blocks are
08:22 < kcalvinalvin> Or I guess you could copy only the index/ dir and point to that
08:23 < adiabat> hmm
08:23 < adiabat> on mainnet, it's 91M
08:23 < adiabat> so could just copy the whole thing into RAM and read from there
08:24 < adiabat> the OS doesn't care about levelDB being picky, it'll let you copy the files even though bitcoind has a lock on em
08:26 < ThomasV> kcalvinalvin: why are you not using bitcoind's RPC to read the blocks?
08:27 < adiabat> we're not
08:27 < adiabat> we can't...
08:27 < adiabat> we read blocks directly from blk0000_.dat and rev0000_.dat
08:27 < ThomasV> why can't you?
08:28 < adiabat> there's no RPC that provides the rev data
08:28 < ThomasV> what is in the rev data?
08:28 < adiabat> in the blocks folder, there's rev000xx.dat
08:28 < adiabat> those have all the utxos consumed in a block, and it mostly corresponds to the blocks in block0000xx.dat
08:29 < adiabat> but not *quite*, which is annoying :)
08:29 < adiabat> so the rev000xx.dat files have a big part of the data we need to build the utxo proofs, and it's already mostly arranged the right way
08:30 < ThomasV> long term you'll probably have to use RPC, so you need'll to parse the transactions
08:30 < ThomasV> you'll need
08:31 < adiabat> well maybe long term the code is merged in to bitcoind or something
08:31 < adiabat> right now there is no RPC that gives any of that data out, so we'd have to introduce new ones into bitcoind which seems hard
08:32 < adiabat> the rev files are kind of interesting because they're not well known, and they take up a ton of space
08:32 < adiabat> also if you prune and don't have the old rev files, you can't reorg back before that
08:32 < kcalvinalvin> Hopefully it gets into Bitcoin Core before I'm 30
08:32 < kcalvinalvin> 7 years *might* be enough
08:32 < adiabat> heh, or btcd
08:32 < ThomasV> introducing a new RPC sounds easier than merging the whole thing into bitcoin core
08:32 < adiabat> yeah maybe...
08:33 < adiabat> how does btcd do blk/rev files...?
08:33 < kcalvinalvin> all the data that would be in rev is thrown all into leveldb
08:33 < adiabat> oh that's worse
08:33 < adiabat> that's... substantially worse, weird
08:33 < kcalvinalvin> Prob one of its bottlenecks
08:33 < kcalvinalvin> Well it does keep all the utxos in memory so...
08:34 < adiabat> yeah I would imagine.  the blk/rev files are definitely optimized for writing
08:34 < kcalvinalvin> Also, I think a lot of the genproofs time is spent on reading blocks
08:34 < adiabat> like it just dumps them on to disk in whatever order it feels like, which is why we get to have fun reading them 
08:34 < kcalvinalvin> Cuz the test I've written does nothing but read blocks from disk and it's sooooo slow
08:34 < adiabat> reading as in deserialize?
08:34 < kcalvinalvin> No not even that
08:34 < ThomasV> btw, how long does genproof take currently on mainnet?
08:35 < adiabat> genproof should only take a few hours right?
08:35 < adiabat> it's faster than syncing up, or should be
08:35 < adiabat> because genproofs doesn't do any signature checking
08:35 < kcalvinalvin> Haven't benched the *whole* mainnet in a while but I assume it takes a few hours
08:36 < kcalvinalvin> The good news is that there's plenty of optimizations left to do
08:36 < adiabat> yeah I don't think anything has made genproofs slower.  well #144 might heh.  hopefully not
08:41 < kcalvinalvin> oh I should mention leveldb from core is only used during the offsetfile generation
08:42 < kcalvinalvin> It's not needed after that
08:42 < adiabat> right
08:43 < kcalvinalvin> So *technically* you can run bitcoind while running genproofs
08:43 < adiabat> ah OK there's a rev offset as well, cool
08:43 < adiabat> I think a good way to do it would be, have genproofs like
08:43 < adiabat> use... os.Copy? or ioutil or something to read the whole folder into ram, then tell levelDB to open that copy
08:44 < adiabat> then build the offset file
08:44 < adiabat> then delete the copied db from ram
08:44 < adiabat> I mean it's fine now, we can just say to stop bitcoind when you start this
08:44 < adiabat> but that might be an effective way to not interrupt bitcoind, since the database we're looking at is so small
08:45 < adiabat> and the actually data we need is even smaller, it's like a few megabytes
08:52 < kcalvinalvin> ok yeah I'll look at doing that
09:10 -!- ThomasV [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 272 seconds]
10:18 -!- ThomasV [~thomasv@unaffiliated/thomasv] has joined #utreexo
10:25 < rjected> adiabat: I asked jcv to take a look at the btcd pr, he requested changes
10:44 -!- rafalcpp [~racalcppp@ip-178-214.ists.pl] has quit [Ping timeout: 260 seconds]
11:25 -!- ThomasV [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 272 seconds]
11:29 < adiabat> oh cool, yeah that's easy to fix
11:29 < adiabat> also rjected - what do I do to get your imports working with go 1.12?
11:30 < adiabat> maybe should ask on direct or search more about go modules...
11:37 < adiabat> kcalvinalvin: on #144 I get a panic leveldb: not found
11:37 < adiabat> after it finished the blk offsetfile... can put in github
11:49 -!- rafalcpp [~racalcppp@ip-178-214.ists.pl] has joined #utreexo
11:59 < adiabat> kcalvinalvin: I'm on 206fe55d8a1bf529839db633de0434e141db01bd
12:00 < adiabat> also it might be a path thing, I'm giving it a datadir
12:03 < adiabat> no it's not a datadir thing, I made a ~/.bitcoin and put it there and ran without datadir but same thing
12:32 -!- ThomasV [~thomasv@unaffiliated/thomasv] has joined #utreexo
12:48 -!- rafalcpp [~racalcppp@ip-178-214.ists.pl] has quit [Ping timeout: 264 seconds]
14:08 < rjected> adiabat: I put a commit on top of the current? utxoview branch in github.com/rjected/utreexo/tree/sad-sed
14:08 < rjected> I'm able to build / test / etc with that
14:10 < rjected> oh might need to update this actually
14:10 < rjected> but yeah you can sorta see what it's doing in the commit
14:19 -!- ThomasV [~thomasv@unaffiliated/thomasv] has quit [Ping timeout: 246 seconds]
14:21 < rjected> yeah okay updated, now it points to the current HEAD of rjected/btcd
14:31 < rjected> just check that out, let me know if you prefer that to replacing all btcd imports in rjected/btcd. cons are that we have to update the go.mod pseudo-version in utreexo every time we want to use a feature added in btcd
14:43 -!- rafalcpp [~racalcppp@ip-178-214.ists.pl] has joined #utreexo
15:26 < adiabat> hm... ok maybe can go over the options on call tomorrow?
15:28 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection]
15:29 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #utreexo
17:02 -!- dergoegge [~dergoegge@196.240.57.124] has joined #utreexo
17:19 < rjected> yeah let's talk about it then
17:25 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
17:51 -!- dergoegge [~dergoegge@196.240.57.124] has quit [Quit: leaving]
19:07 -!- rafalcpp [~racalcppp@ip-178-214.ists.pl] has quit [Ping timeout: 264 seconds]
19:54 -!- ThomasV [~thomasv@unaffiliated/thomasv] has joined #utreexo
20:07 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has joined #utreexo
21:32 -!- ThomasV [~thomasv@unaffiliated/thomasv] has quit [Read error: Connection reset by peer]
21:44 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
22:04 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has joined #utreexo
22:41 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
22:51 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has joined #utreexo
22:59 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
23:57 -!- ThomasV [~thomasv@unaffiliated/thomasv] has joined #utreexo
--- Log closed Tue Jun 09 00:00:46 2020