--- Day changed Wed Dec 11 2019
09:16 < adiabat> OK so ideas for the TTLdb not using an actual DB
09:17 < adiabat> (because part of why bitcoin core people like utreexo is that it doesn't need leveldb, so if we're using a lot of levelDB that's not great)
09:17 < adiabat> we have rev___.blk data, but that doesn't quite have what we want
09:17 < adiabat> say we're looking at block 17
09:17 < adiabat> the blk___.dat tells us the whole block
09:18 < adiabat> the rev___.dat for block 17 tells us about all the utxos destroyed
09:18 < adiabat> including how old those UTXOs were
09:18 < adiabat> which is *close* to what we want for TTL values
09:19 < adiabat> what we want is somewhere, for every UTXO created, we want to know how long it lives
09:19 < adiabat> that's the rev data but not in the right place; it tells the TTL data but when the utxo dies; we want to know it when it's born
09:20 < kcalvinalvnn> TTL= Time To Live. Basically how long a tx lasts until it's spent
09:20 < adiabat> the rev data doesn't tell us *where* in the block a utxo came from, it just tells us which block
09:20 < adiabat> yeah... so the simple way to do it is make a key:value store like leveldb
09:20 < adiabat> and the key is the 36 byte outpoint (txid:index)
09:20 < adiabat> and value is TTL
09:21 < adiabat> and we can make those just from the rev data 
09:21 < adiabat> but... then we're using levelDB
09:21 < adiabat> if we want to use a flat file though, we can; we know how many utxos are created in a block as soon as we see the block
09:22 < adiabat> we can then assign a ... 40 byte slot for every utxo created 
09:23 < adiabat> and when we get to rev data deleting a utxo, seek to that block and write the outpoint and TTL data in the first empty 40 byte slot
09:23 < adiabat> kindof wastes a lot of space, and then it's out of order
09:23 < adiabat> but only out of order within a block, whcih is pretty small, like a few thousand entries
09:24 < adiabat> then once we're done we could take a pass through and sort them.
09:25 < adiabat> or leave it unsorted, when you want to serve TTL values just load em all into a hashmap and then it gets auto-sorted
09:26 < adiabat> it's pretty small, less than 100KB, and we might end up using doing sorting and stuff anyway to the TTL data (like sort by TTL to give all the short-lived utxos)
09:27 < adiabat> it's a bunch more space on disk though; instead of 4 bytes for every output in a block, it's more like 40 (though could probably have truncated txids)
09:28 < kcalvinalvnn> I can see how the implementation would work. Buuuut seems a lot more messy?
09:28 < adiabat> yeahhhh levelDB is easier
09:28 < adiabat> also we have the code to do it already
09:30 < adiabat> another option is just skip all the TTL stuff for now and run without it.. it's an optimization (a pretty big one but there are other ways to do it)
09:32 < kcalvinalvnn> I guess we could cache like Bitcoin Core?
09:32 < adiabat> you can do look-behind instead of look-ahead
09:33 < kcalvinalvnn> How did the Core people feel about the caching system? Because technically it *is* an attack vector
09:33 < adiabat> which means you just keep all the leaves from the last, say, 20 blocks
09:33 < adiabat> look-ahead is much nicer though
09:33 < adiabat> you never have to eject 
09:33 < adiabat> and you don't have to keep track of how long something's been there
09:34 < adiabat> oh, TTL based caching..?  Yeah I mean the attack is to keep a lot of utxos around a long time, which is already a decent attack on the utxo set...
09:35 < kcalvinalvnn> At the same time it *is* a lame attack too
09:36 < adiabat> Yeah I'm not worried that people will like, change a large portion of all transactions just to make utreexo a bit slower
09:37 < kcalvinalvnn> Did you get the chance to test the txottlgen for the full blockchain?
09:37 < kcalvinalvnn> I can ditch the pipedream for now and just change the txottlgen to bytes from string which would help a lot
09:37 < adiabat> mainnet?
09:38 < kcalvinalvnn> well both I guess
09:38 < adiabat> or testnet?  I can try that today
09:38 < adiabat> wait what would change to bytes?
09:39 < kcalvinalvnn> I never touched ttl stuff after the resume thingy
09:39 < kcalvinalvnn> so it's still making a ttl.*.txos file that's in ascii
09:40 < adiabat> hm yeah we should just get rid of that entirely
09:40 < adiabat> runIBD can just read from the ttldb
09:40 < adiabat> maybe slightly slower than having a flat file for it i guess
09:40 < kcalvinalvnn> Ok just do math on the fly then?
09:41 < adiabat> yeah that's probably not a big bottlneck... well, maybe it is?  I'm not sure
09:41 < adiabat> the math part is quick, it's the lookups that might be slow
09:41 < adiabat> gotta look up a couple thousand entries in the ttldb each block
09:42 < kcalvinalvnn> reading ttl data should be fine I think. It was choking on deserializing blocks last time
09:42 < adiabat> i'd say leave the DB for now and have runIBD read the db directly, and if that seems slow we can have another pass
09:42 < adiabat> a whole pass just to make a flat txottl file seems like a bit of a waste
09:42 < kcalvinalvnn> ok cool. Lots of code gone
09:43 < adiabat> it's probably be really quick though...?  except not really, we'd still have to read / deserialize the whole block...
09:44 < kcalvinalvnn> Less deserialization the better. Reading from ascii files was faster
09:44 < adiabat> but yeah I think to start just have runIBD do it, way less code and hopefully not much slower
09:46 < kcalvinalvnn> Ok so: 1) get rid of txottl 2) have ibdsim read directly from leveldb?
09:47 < kcalvinalvnn> gah also, if leveldb is still present, is it a no go?
09:48 < kcalvinalvnn> I think there is value even without the removal of leveldb
09:55 < kcalvinalvnn> Well, I'll work on those two things for now
10:18 < adiabat> yeah I mean it's fine to still have levelDB for now
10:19 < adiabat> that's more of a long-term goal, for bitcoind as well
10:19 < adiabat> like even if you get rid of the utxo set there's still serveral uses of leveldb
--- Log closed Thu Dec 12 01:11:45 2019
--- Log opened Thu Dec 12 01:12:30 2019
01:12 -!- kanzure [~kanzure@unaffiliated/kanzure] has joined #utreexo     
01:12 -!- Irssi: #utreexo: Total of 17 nicks [0 ops, 0 halfops, 0 voices, 17 normal]     
01:25 -!- Irssi: Join to #utreexo was synced in 818 secs