--- Log opened Fri Mar 13 00:00:25 2020
02:05 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:590f:c388:d48d:76c1] has joined #utreexo
02:37 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:590f:c388:d48d:76c1] has quit [Remote host closed the connection]
04:37 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:590f:c388:d48d:76c1] has joined #utreexo
04:38 -!- nick_fre_ [~nick_free@2001:16b8:30e2:a200:5cc8:95f2:255f:4ac7] has joined #utreexo
04:41 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:590f:c388:d48d:76c1] has quit [Ping timeout: 240 seconds]
04:42 -!- nick_fre_ [~nick_free@2001:16b8:30e2:a200:5cc8:95f2:255f:4ac7] has quit [Ping timeout: 240 seconds]
05:22 -!- nick_freeman [~nick_free@92.116.142.66] has joined #utreexo
05:25 -!- nick_freeman [~nick_free@92.116.142.66] has quit [Remote host closed the connection]
05:37 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:5ca5:df80:f3a6:e817] has joined #utreexo
05:37 -!- nick_fre_ [~nick_free@2001:16b8:30e2:a200:d852:2a3b:82ee:7a5e] has joined #utreexo
05:41 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:5ca5:df80:f3a6:e817] has quit [Ping timeout: 240 seconds]
06:21 -!- nick_fre_ [~nick_free@2001:16b8:30e2:a200:d852:2a3b:82ee:7a5e] has quit [Remote host closed the connection]
06:24 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:d852:2a3b:82ee:7a5e] has joined #utreexo
06:44 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
07:11 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:d852:2a3b:82ee:7a5e] has quit [Remote host closed the connection]
09:11 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:d852:2a3b:82ee:7a5e] has joined #utreexo
09:12 -!- nick_fre_ [~nick_free@2001:16b8:30e2:a200:2058:24aa:8f4e:aca5] has joined #utreexo
09:16 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:d852:2a3b:82ee:7a5e] has quit [Ping timeout: 272 seconds]
09:17 -!- nick_fre_ [~nick_free@2001:16b8:30e2:a200:2058:24aa:8f4e:aca5] has quit [Ping timeout: 256 seconds]
09:25 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has joined #utreexo
09:48 < adiabat> ysangkok: ah sorry, I explained it to kcalvinalvin on jitsi but didn't write anything on github, can write & close
09:49 < adiabat> ysangkok: for knowing "where" a hash is, there's the forest.positionMap
09:49 < adiabat> give it a (truncated) hash, and it will give the uint64 position back
09:49 < adiabat> which... is annoying and means that the bridge nodes will probably need leveldb or something like that.
09:50 < adiabat> right now we can regenerate the whole map just by reading through the base of the tree
09:58 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:fc86:2abf:b899:92f3] has joined #utreexo
10:09 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:fc86:2abf:b899:92f3] has quit [Remote host closed the connection]
10:39 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
10:43 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Ping timeout: 245 seconds]
12:38 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
12:55 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Remote host closed the connection]
12:55 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
12:55 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Remote host closed the connection]
12:55 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
13:05 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Remote host closed the connection]
13:10 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
13:14 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Remote host closed the connection]
13:16 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
13:21 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Remote host closed the connection]
13:22 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
13:23 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Remote host closed the connection]
13:23 -!- nick_fre_ [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
13:31 -!- nick_fre_ [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Remote host closed the connection]
13:34 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has joined #utreexo
16:02 < ysangkok> adiabat: do you think it would make sense to have that in the public API? i could make a PR with a public method that gives you a ([Hash]map uint64) for a ([Hash]map struct{})
16:05 < ysangkok> oops, messed up the syntax there, i meant map[Hash]uint64 and map[Hash]struct{}
16:25 -!- gojiHmPFPN [~textual@c-73-47-220-190.hsd1.ma.comcast.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
17:32 -!- pigeons_ [~pigeons@androzani.sysevolve.com] has joined #utreexo
17:34 -!- pigeons [~pigeons@androzani.sysevolve.com] has quit [Ping timeout: 240 seconds]
17:34 -!- ysangkok [janus@anubis.0x90.dk] has quit [Ping timeout: 240 seconds]
17:34 -!- ysangkok [janus@anubis.0x90.dk] has joined #utreexo
19:23 -!- nick_freeman [~nick_free@2001:16b8:30e2:a200:8d68:62c4:5660:770e] has quit [Remote host closed the connection]
19:24 < adiabat> ysangkok: yeah I'm not sure!  I agree the current setup isn't what we want, but there are a couple ways to go about fixing it-
19:24 < adiabat> right now the main methods that forest provides are Modify(adds []LeafTXO, dels []uint64)
19:25 < ysangkok> you mean there are multiple desirable APIs?
19:25 < adiabat> well I mean I don't know the best way to do it :)
19:25 < adiabat> ProveBlock(hs []Hash) takes hashes, and returns a block proof
19:25 < adiabat> but Modify takes in hashes and uint64
19:26 < adiabat> This works in that we can call forest and have it keep the positionMap totally internal and external callers don't have to worry about iot
19:26 < adiabat> *it
19:26 < adiabat> but there's 2 problems with that
19:26 < adiabat> first, what if we want to change it to levelDB or something other than an in-ram map, save it to disk, etc
19:27 < adiabat> second, we actually need more than what ProveBlock gives: the BlockProof struct it returns doesn't contain anything other than proofs
19:27 < adiabat> it also needs to contain TxInUndo data
19:27 < ysangkok> to make a proper proof?
19:28 < ysangkok> you'd need more for which use case?
19:28 < adiabat> for bitcoin to know what the utxo data is
19:28 < ysangkok> aah
19:28 < adiabat> I'm leaning towards keeping the utreexo library non-bitcoin specific
19:28 < adiabat> like anyone could use it to throw whatever hashes they want in, and then get inclusion proofs for those hashes out
19:29 < adiabat> but in bitcoins case, we need not just a proof that the utxo for an input exists
19:29 < adiabat> but also what it is, like the pubkey script and the amount of satoshis
19:30 < ysangkok> so when you talk about leveldb, you talk about just storing that in a kv store?
19:30 < ysangkok> so not actually in the forest?
19:30 < adiabat> anyway if it's easier for something we can definitely make the positionMap into a PositionMap and public
19:30 < adiabat> right what the positionMap would turn into is
19:30 < adiabat> map [Hash]someStruct
19:30 < adiabat> where the someStruct would be, I guess
19:31 < adiabat> position uint64, utxoData
19:31 < ysangkok> well i am just trying to refine the forest itself because i am also seeing it in this light of "general datastructure", i just asked the question because i was wondering about the proper external interface
19:31 < adiabat> where utxoData has a script []byte, amount int64, height int32
19:31 < adiabat> and maybe that's all
19:31 < adiabat> right... for the general data structure, I think we should hide the positionMap...
19:32 < adiabat> but I'm not even sure about that actually
19:32 < adiabat> like ProveBlock gives you the proof you need
19:32 < adiabat> so an external caller shouldn't need to directly worry about positions, or even know that there are positions
19:33 < ysangkok> so in Java generics terms, it is actually Accumulator<Hash,Tuple<Script,Amount,Height>>, 
19:33 < adiabat> it would be for bitcoin, yes
19:33 < adiabat> the accumulator itself just stores hashes
19:34 < adiabat> so there's another part that needs to keep track of the utxo data (the script/amount/height)
19:34 < adiabat> basically the preimage of all those hashes
19:34 < ysangkok> yeah, makes sense
19:34 < adiabat> the tricky thing is in that in a lot of cases, you don't actually need the script data, which is the biggest part
19:35 < adiabat> most scripts are p2pkh or p2sh
19:35 < adiabat> or the segwit equivalents
19:35 < adiabat> and in that case, you basically get the pre-image of the script in the signature / witness
19:35 < ysangkok> hmmm are wallet descriptors applicable here?
19:35 < ysangkok> they are basically generalizations of scripts, iirc
19:36 < adiabat> I don't think wallet descriptors would work here since we have to be super general
19:36 < adiabat> it needs to support any weird script we don't even know about yet
19:36 < ysangkok> yeah, but still optimize for the common case. it is a union: p2pkh OR p2sh OR some-custom-script
19:37 < adiabat> right, so in any p2pkh / p2sh / p2wpkh / p2wsh case
19:37 < adiabat> we don't have to save the script, since it will be apparent at spend time in the block itself
19:37 < adiabat> we also don't have to save the outpoint, since that will also be in the spending tx
19:37 < adiabat> the hash itself will commit to those things though
19:38 < adiabat> so in most cases, the extra data proveBlock needs to return is just
19:38 < adiabat> amt int64, height int32
19:38 < adiabat> so either 12 bytes, or if we are OK with variable lenght you can compress those
19:39 < adiabat> problem though is in some cases we need lots of data for the script
19:40 < ysangkok> when would the heights/amounts/scripts enter utreexo.Forest? if they are to be returned by ProveBlock
19:41 < adiabat> that's the thing.  Should forest know about them?  if it does, that makes it very bitcoin specific
19:41 < adiabat> maybe there's just an extraData field associated with each add...?
19:41 < adiabat> or maybe a separate thing in cmd keeps track of all that
19:41 < ysangkok> well, if you see the Accumulator as some kind of Map, it can be generic in many languages (not so easy in go :P)
19:42 < ysangkok> still applicable to other things than bitcoin even if it has the heights/amounts/scripts in the api signatures, cause they would be generic arguments in other languages
19:42 < adiabat> ehhh it is kindof a map, yeah but... in practice not as simple to add and remove :)
19:43 < ysangkok> yeah ok, i don't know what to call it. but it is an API that generalizes over multiple datatypes
19:43 < adiabat> well I just mean if forest.Modify takes in (adds []Leaftxo, dels []uint64) and LeafTxo is more than just a hash
19:43 < adiabat> hm I guess it already says "leafTXO" which is bitcoin stuff :)
19:43 < adiabat> also... Duration is unused...? i think
19:44 < ysangkok> hmmm i think i got crashes when i had negative durations...
19:44 < adiabat> hmm it's used but not in forest itself
19:44 < ysangkok> anyway, i guess we can generalize later if it makes things easier
19:45 < adiabat> yeah it's just a matter of keeping layers / things separate
19:45 < adiabat> hm.  this is making me think though
19:45 < adiabat> maybe the proofMap should just have an extraData part, which is just []byte
19:46 < adiabat> and it can give that byte slice back when ProveBlock is called
19:46 < adiabat> that would be enough for our use now
19:46 < ysangkok> oh, that seems straightforward
19:46 < adiabat> I'm not sure if that's the best way long term but...
19:46 < adiabat> we don't *actually* have any other uses for this other than bitcoin yet
19:47 < adiabat> and if it turns out someone else uses it, then cool, and if they don't need the extraData part they can just leave it blank, no big loss
19:47 < ysangkok> yeah. but how much data is that, is it ok to store it in memory for starters?
19:47 < adiabat> heh yeah in bitcoin's case it is, at least for now
19:48 < adiabat> the idea being that a bridge node should have 8GB of ram or so
19:48 < adiabat> but ... maybe it'd be nice to run a bridge node with minimal ram
19:48 < adiabat> ForestData is now an interface so it *can* run off disk, though it's a bit slower
19:49 < adiabat> positionMap could also be made that way I guess
19:49 < adiabat> just have levelDB... I was really hoping to get *away* from levelDB but for bridge nodes it doesn't seem possible
19:50 < adiabat> for pollard though, no problem :)
19:50 < ysangkok> btw https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md#features claims that descriptors also cover raw hex scripts
19:51 < adiabat> yeah I mean.. I supposed we could convert the scripts into this descriptor format
19:52 < adiabat> but this seems more for user interface
19:52 < ysangkok> ok
19:52 < adiabat> like we don't know any of the derivation data 
19:52 < ysangkok> oh true
19:52 < ysangkok> yeah hard to compress random looking data, i guess...
19:53 < adiabat> so yeah possible but not sure it adds anything
19:53 < adiabat> I think in most cases it can be very small
19:53 < ysangkok> ok, so if i understand correctly, you think it would be best to do leveldb, so that the node can run on smaller memory systems?
19:53 < adiabat> sigh... yeah....
19:54 < adiabat> so disappointing heh
19:54 < adiabat> but yes, it seems it can't be regenerated from the base of the tree
19:54 < adiabat> so maybe I can just get rid of the map entirely and switch it to leveldb...
19:55 < ysangkok> you prefer leveldb over bbolt?
19:55 < adiabat> levelDB, it seems a lot faster
19:55 < adiabat> hmm.  there's really only one place it's called from
19:55 < adiabat> well 2 or 3
19:56 < adiabat> addv2, and swapNodes
19:56 < adiabat> the rest are not really important
19:56 < adiabat> or at least wouldn't apply if it were a db
19:56 < adiabat> those are the writes I mean, proveBlock does reads
19:57 < adiabat> so yeah really only a few places to replace it with db calls
19:57 < adiabat> might be less code as we could get rid of RestoreForest stuff
19:57 < ysangkok> interesting, yeah, doesn't seem to hard. you are talking about accesses to proofMap, right?
19:58 < adiabat> yeah
19:58 < adiabat> the extradata would be like... 1 byte for a flag/type
19:59 < adiabat> and then if the flag is something known, the only data is amount & height
19:59 < adiabat> if the flag is unknown, it's the whole pkscript
19:59 < adiabat> (also with amount & height)
20:00 < adiabat> and we can use all the weird tricks bitcoin core does, like isCoinbase is the LSB of height
20:00 < adiabat> and you have to >>1 all the heights after checking coinbaseness, etc
20:00 < ysangkok> so if it is p2pkh for example, no script at all is stored?
20:00 < adiabat> yeah, don't need to
20:00 < adiabat> when you see the signature, it includes the pubkey
20:01 < ysangkok> aaaaah ok
20:01 < adiabat> and can hash that to get the script back
20:01 < adiabat> so like, everything still goes into the leaf
20:01 < ysangkok> so gonna break for schnorr :P
20:01 < adiabat> hmmmmm does it?
20:01 < adiabat> shoot
20:01 < ysangkok> schnorr does not have pk recovery iirc
20:01 < adiabat> because they commit to P in e
20:01 < adiabat> and I always didn't like that
20:02 < adiabat> bah.  well.  can throw in a hash of the pubkey instead of the whole thing
20:02 < adiabat> hm.  actually.  any time it's an unknown script type can just use a hash of it
20:03 < ysangkok> ah cool, yeah
20:03 < adiabat> well with schnorr that doesn't save space I guess
20:03 < ysangkok> so that sets an upper bound on it, at least. it seems
20:03 < adiabat> unless we use hashes shorter than 32 bytes
20:04 < adiabat> which I'm pretty sure is safe
20:04 < adiabat> but... I'm not going to push that
20:06 < adiabat> shoot, didn't realize schnorr breaks that...
20:08 < adiabat> and it would totally work if it was hash(R || m) instead of hash(R || P || m)
20:11 < adiabat> .. or maybe not; it might be complicated.  you need to know R, s, m in order to compute the potential P
20:11 < adiabat> but m could have other pubkeys from other inputs in the sighash
20:11 < ysangkok> oh tricky :O
20:11 < adiabat> anyway whatever, another 32 bytes of data for schnorr sigs
20:12 < adiabat> also ... no actually, can't just take a hash of arbitrary length scripts...
20:12 < adiabat> I think you have to store the whole thing
20:13 < adiabat> there's hardly any large scripts so doesn't matter in practice... but it would be nicer to have a fixed length
20:13 < ysangkok> if using leveldb it probably won't be a problem, i think
20:17 < ysangkok> so you're saying we can't hash a output script that is not p2wsh or p2sh?
20:17 < ysangkok> because the script will be needed for verification?
20:20 < ysangkok> bip-schnorr says they need to commit to P for musig
20:23 < adiabat> yeah I guess for musig need that...
20:24 < ysangkok> i am not sure i understand the kind of node you wanna end up with
20:24 < ysangkok> will it have script execution?
20:24 < ysangkok> the signatures sign a hash of the script...
20:24 < adiabat> if the pre-image of the output script appears in the signature, we don't needs to save the script
20:24 < adiabat> yeah, all scripts and signatures can be verified
20:25 < adiabat> you can get back the output script from looking at the signature
20:25 < adiabat> for p2pkh, it's <sig> <pubkey>
20:25 < adiabat> just hash the pubkey in hash160
20:26 < adiabat> and you've got the original output script
20:26 < ysangkok> aah, so by signature, you mean scriptSig
20:26 < adiabat> yeah
20:26 < adiabat> arg too many terms
20:26 < adiabat> or witness, same idea
20:27 < adiabat> same with p2sh / p2wsh, you take the whole script given and hash it to get back to the scriptPubKey
20:27 < adiabat> it could be wrong, but then it won't match the leaf
20:27 < adiabat> since the whole scriptPubKey goes into the leaf hash
20:28 < ysangkok> yeah, better than electrum :)
20:28 < adiabat> in terms of validation?  yeah the idea is it's a full node
20:28 < ysangkok> yep yep
20:28 < adiabat> and if some day every node & wallet uses it, then you don't need bridge nodes!
20:29 < adiabat> (which will basically never happen but hey)
20:30 < ysangkok> but all those raw scripts, are they even standard? if they are not, maybe you could get away with not supporting them at first?
20:30 < ysangkok> yeah, a beautiful dream... and still more practical than interactive tx construction as with mimblewimble
20:31 < ysangkok> utreexo could ship with the preimages of the raw output scripts that are in the current blockchain? but ugly
20:37 < ysangkok> i guess you can't compromise on "full-node", so you can't just assume standardness...
20:47 < adiabat> yeah even though there aren't many raw scripts, can't really skip them
20:47 < adiabat> also there are few enough that it will take up hardly any space
20:48 < adiabat> yeah the reason I don't think you can ever get rid of bridge nodes is that there will always be SPV wallets, web wallets, things like that
20:48 < adiabat> but who knows, maybe some day a bridge node would be something like a node with an address index, like an electrum server
20:49 < adiabat> another option is that it seems not too hard to have a partial bridge node
20:51 < ysangkok> yeah, it has lots of potential. though the guy who runs the electrumx server project is an altcoiner...
20:52 < adiabat> the electrum model... isn't great.  people just send their whole address list to unknown servers
20:52 < ysangkok> totally agreed :)
20:53 < ysangkok> but it would be nice to have those people run bridge nodes :P
20:53 < ysangkok> so i remember the term CSN, compact state node, is that what you just described?
20:54 < adiabat> the CSN is the node that verifies but doesn't have the whole forest
20:54 < adiabat> (basically pollard in the current code)
20:54 < ysangkok> i was thinking maybe it would be good to have a glossary in the readme
20:55 < adiabat> yeah... and to change some names
20:55 < ysangkok> which names need changing?
20:57 < ysangkok> also, you said "for pollard though, no problem :)" why is it no problem for pollard?
20:57 < ysangkok> afaik pollard is just forest, but sparse?
20:58 < adiabat> pollard doesn't have a positionMap at all
20:58 < adiabat> it doesn't have a ProveBlock method
20:58 < adiabat> otherwise it's like forest but sparse and pointers instead of having all the data linearly
20:59 < adiabat> so pollard doesn't need levelDB or anything
21:00 < ysangkok> ah ok, so pollard is not for bridge nodes?
21:00 < adiabat> right, pollard can't be a bridge node
21:01 < ysangkok> ok, so CSN uses pollard, bridge uses forest
21:01 < adiabat> if you wanted to make some partial bridge node, I don't know, can think about that later
21:01 < adiabat> for now it seems fine to not worry about that
21:01 < ysangkok> okay
21:01 < adiabat> also if it becomes hard to run a bridge node, that means the utxo set is so big that it's hard to run a regular full node
21:02 < adiabat> since a bridge node has at worst ~2X the data of a regular full bridge node
21:03 < ysangkok> the paper talks about that tradeoff, where on that curve is the current implementation?
21:03 < adiabat> oh the curve thing, like caching vs download?
21:03 < ysangkok> yes
21:03 < adiabat> that's dependent on how much the pollard stores
21:04 < adiabat> when you call modify, the leafTXOs you give it
21:04 < adiabat> have the "remember" bool
21:04 < adiabat> that says whether to store the leaf or not
21:04 < ysangkok> aaah ok. so for forest, remember is not used?
21:04 < adiabat> right, it remembers everything no matter what
21:05 < ysangkok> things are starting to make sense :D
21:05 < adiabat> the problem is there's duration and remember but they shouldn't both be in the same place
21:05 < adiabat> a while ago I had pollard itself making the call on whether to remember or not
21:06 < adiabat> if you tell pollard to remember everything, it won't need any proofs
21:06 < adiabat> so pollard.overWire will be 0
21:06 < adiabat> (hopefully exactly 0...?)
21:06 < adiabat> and if you have remember = false on everything, it will only keep the tree roots
21:06 < adiabat> and need to download a lot of proofs
21:07 < ysangkok> yeah ok, good to finally understand where in the code those things from the paper are
21:08 < adiabat> sure!  Also I have lots of things to change
21:08 < adiabat> maybe should put leveldb in to forest...
21:08 < adiabat> shouldn't be hard.  mostly have been avoiding it because I was hoping to figure out some other way to do it
21:08 < ysangkok> i don't know about bitcoin core's tricks regarding coinbase, i can't do that :O
21:08 < adiabat> I had this whole thing i started with a prefix tree...
21:09 < adiabat> yeah bitcoin core compresses utxos for storage in levelDB
21:09 < adiabat> not compresses like gzip, but lots of weird bitcoin-specific things
21:09 < adiabat> I think getting the same serialization as core is good because: 1- they seem to know what they're doing generally :)
21:10 < adiabat> 2- could be better for compatibility, eventually get it in to core
21:10 < ysangkok> does that mean we need to link with core to test?
21:10 < ysangkok> that would require some bindings...
21:12 < ysangkok> or maybe it is sufficient to open their leveldb and assert stuff
21:13 < adiabat> yeah integration with core is ... I'm not sure
21:13 < adiabat> one intermediate step that might be not too hard
21:13 < adiabat> is a kind of c++ ValidateBlock() call
21:13 < adiabat> the go code could call out to
21:14 < adiabat> so we'd get the same script execution and signature checking as c++ bitcoin core is using
21:14 < ysangkok> i wonder if libbitcoinconsensus has that
21:15 < adiabat> but getting a CSN into core is trickier because the DB is all over the place in the code, so pulling out the DB is a big code change
21:15 < adiabat> utreexo kindof makes libconsensus work
21:15 < adiabat> until now the problem has been that the DB is a critical part of the consensus code
21:16 < kcalvinalvin> It's a huge change :p
21:16 < adiabat> heh yeah
21:16 < ysangkok> looks like somebody has libbitcoinconsensus bindings to go: https://github.com/piotrnar/gocoin/blob/master/tools/verify_tx.go
21:16 < adiabat> adding bridge node functionality to core is not as big a change, but also not as exciting
21:16 < adiabat> since you could have bridge nodes that aren't even verifying / full nodes
21:17 < adiabat> that just store the proofs and stuff without knowing what any of it means
21:17 < adiabat> and the CSNs do all the validation
21:17 < kcalvinalvin> ysangkok the problem was that the libconsensus stuff calls leveldb *a lot*
21:18 < ysangkok> oh, so it would be totally dependent on having a synced bitcoin node?
21:18 < adiabat> what would be dependent?
21:19 < ysangkok> i understood calvin as saying that to execute a bitcoin script, you'd have to call libbitcoinconsensus but that would need the bitcoin leveldb, you can't just provide all the necessary data in the function parameters
21:19 < adiabat> currently yeah
21:20 < adiabat> but a new / better libconsensus or similar code could work
21:20 < adiabat> kcalvinalvin was looking at the btcd transaction verification code
21:21 < kcalvinalvin> oh cool. 
21:22 < kcalvinalvin> I'll push the rev stuff before tuesday. Almost done now
21:22 < adiabat> cfields_ mentioned looking at code that c++ part but might be tough
21:22 < ysangkok> he is talking about merging that into core?
21:22 < kcalvinalvin> the libutreexo thing?
21:23 < adiabat> no like, making a variant of libconsensus
21:23 < adiabat> basically a c++ function that is bitcoin core's block/transaction/script verification
21:23 < adiabat> but without levelDB
21:23 < ysangkok> sounds basically impossible to get merged into bitcoin core :O
21:23 < adiabat> such that all the TxInUndos are given as function arguments instead of looked up as needed
21:24 < adiabat> initially that function wouldn't be in bitcoin core
21:24 < adiabat> but we could call it from go
21:24 < adiabat> and then have a good idea that we're getting the same consensus as core code
21:24 < kcalvinalvin> The rough setup I have right now is leaving out all the protocol rules
21:25 < adiabat> eventually though, there's definitely desire to have a levelDB-less node
21:25 < adiabat> and it's a big change, but I don't think any bigger than say assumeUtxo
21:25 < kcalvinalvin> Doesn't the c++ libconsensus code call the chainstate here and there?
21:25 < adiabat> yeah, I shouldn't say "libconsensus" as it wouldn't be based on that
21:26 < adiabat> it'd have to be a new function patched together from bitcoin core code that did the same thing
21:26 < adiabat> ValidateBlock([]TxInUndo, MsgBlock) bool
21:26 < kcalvinalvin> hmmmmm...
21:28 < ysangkok> all we need is a usable API. if libbitcoin has a useble API, wouldn't it be fine to start out with that?
21:29 < adiabat> but libconsensus doesn't
21:29 < ysangkok> yeah, and i guess btcd doesn't either?
21:30 < ysangkok> that's why i mentioned a third implementation :P
21:30 < adiabat> btcd doesn't have anything like libconsensus no
21:31 < ysangkok> this doesn't seem to mention any particular db: https://github.com/libbitcoin/libbitcoin-system/blob/master/src/machine/program.cpp :O
21:32 < adiabat> hm I guess I don't even know what libbitcoin does
21:33 < kcalvinalvin> btcd has 5 basic things for validating consensus (besides protocol rules). blockchain.CheckTransactionSanity(), blockchain.IsCoinBase(), blockchain.CheckTransactionInputs(), blockchain.GetSigOps(), and blockchain.ValidateTransactionScripts()
21:33 < kcalvinalvin> That's what I'm currently working off of
21:35 < ysangkok> adiabat: it is aiming to do pretty much everything, i am sure it doesn't do everything well. but i did meet the author and he knew quite a lot of details of bitcoin scripting, so i think the interpreter is complete
21:35 < adiabat> maybe we can use libbitcoin consensus...
21:36 < ysangkok> are you talking about the one in bitcoin core or this one: https://github.com/libbitcoin/libbitcoin-consensus ?
21:36 < adiabat> is it the same code as bitcoin core...? doesn't seem the same?
21:36 < adiabat> yeah like here https://github.com/libbitcoin/libbitcoin-consensus/tree/master/src/clone/script
21:37 < ysangkok> no, not the same as core at all, completely independent
21:37 < adiabat> but supposed to be consensus compatible I guess?
21:37 < ysangkok> yes of course, it is a bitcoin implementation
21:39 < adiabat> in this repo anyway I don't see anything about block validation
21:40 < ysangkok> adiabat: it is in another repo: https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/src/validate/validate_block.cpp
21:41 < adiabat> ok, looking through; the parts that say These checks require chain state, and block state if not under checkpoint.
21:42 < adiabat> ... what's a bucket... hmm
21:43 < ysangkok> hmmm, that is a bad sign, if that is the chain state of core
21:44 < ysangkok> i think buckets is the level of concurrency?
21:44 < adiabat> yeah I don't know where all these things are defined, maybe other repos
21:46 < ysangkok> well, as you linked, i guess they do embed libconsensus
21:46 < ysangkok> so while the interface may be cleaner, it is probably fundamentally not a big change
21:46 < ysangkok> that comment you mentioned is a very bad sign, i guess
21:47 < adiabat> I'm still looking through it
21:47 < adiabat> can't really figure out what it's doing
21:47 < adiabat> there's not much documentation
21:48 < adiabat> the wiki has... a whole bunch of weird stuff in "cryptoeconomics" but not much about how to use the validation libraries
21:48 < kcalvinalvin> oh heh that's just Eric Voskuil
21:48 < ysangkok> the readme of the blockchain repo does say "The default configuration requires libbitcoin-consensus. This ensures consensus parity with the Satoshi client. To eliminate the libbitcoin-consensus dependency use the --without-consensus option. This results in use of libbitcoin consensus checks."
21:49 < ysangkok> that makes it sound like if you use that flag, it should have it's own script execution engine
21:49 < ysangkok> kcalvinalvin: you know him? can't say i know him, though i did meet him
21:49 < adiabat> it seems like it's geared more towards wallet functionality?
21:49 < ysangkok> hmmm i don't know his applications...
21:50 < ysangkok> but if you think it might have promise, i could ask him if he thinks it is easier to use for us
21:50 < adiabat> like if you want to verify a block, what do you call, and what data do you give it?
21:51 < ysangkok> he always talked so much about performance, so it is probably a bunch of callbacks in all those parameters to the validate_block class...
21:51 < ysangkok> so that he can have it async
21:52 < kcalvinalvin> iirc the performance he was talking about was scalability, as in, if you throw more cpus at it, it'll use it
21:52 < adiabat> right but I'm just looking for... what to call
21:52 < adiabat> there's some root validate function that calls the rest?
21:54 < ysangkok> there is validate_block::connect which takes a handler that is called with something called success , maybe that is the callback that is triggered if the block validates?
21:56 < adiabat> maybe... I'm just trying to find where it actually does anything
21:56 < adiabat> like... add up all the fees and checks coinbase tx amount
21:58 < ysangkok> total_input_value: https://github.com/libbitcoin/libbitcoin-system/blob/master/src/chain/transaction.cpp#L782
21:58 < ysangkok> fees is calculated from total_input_value: https://github.com/libbitcoin/libbitcoin-system/blob/master/src/chain/transaction.cpp#L853
21:58 < adiabat> but that's libbitcoin-system
21:59 < adiabat> not blockchain or consensus 
22:00 < adiabat> I dunno this just looks like a complete rewrite of bitcoin core
22:00 < ysangkok> pretty sure it is a complete reimplementation
22:00 < ysangkok> is_valid_coinbase_claim : https://github.com/libbitcoin/libbitcoin-system/blob/master/src/chain/block.cpp#L693
22:01 < adiabat> ok so validation code is spread out among all the repos I guess
22:01 < ysangkok> yeah annoying
22:05 < ysangkok> hard for me to judge what is easiest to integrate, especially since i havn't looked at btcd and core very much
22:05 < adiabat> yeah I dunno, btcd might be easier since it's in go and we can chop it up easier
22:06 < adiabat> also since btcd isn't that heavily used and I think roasbeef is the only maintainer, if we get things working there could probably get it merged
22:06 < adiabat> libbitcoin seems just as hard as bitcoin core
22:06 < adiabat> maybe easier, it's hard to tell what it's doing, there's no real documentation about how to use it as a library
22:07 < ysangkok> what is the current stumbling block with btcd?
22:07 < adiabat> I haven't really looked, I think kcalvinalvin is looking
22:08 < ysangkok> it was great chatting with you, i feel like i understand it a bit more now... off to bed!
22:08 < adiabat> sure, thanks
22:09 < adiabat> I have lots to work on even before merging with a whole node :)
--- Log closed Sat Mar 14 00:00:22 2020