--- Log opened Fri Jan 08 00:00:09 2021
09:04 < adiabat> ja: what do you mean by non-deterministic?  Is there a source of randomness?
09:05 < adiabat> there are definitely other variants of the remove transform that can work; there's lots of freedom and the current transform is pretty much the simplest
09:06 < ja> adiabat: i just mean that my algorithm generates a list of forests, while remtrans implies just one forest. but the remtrans forest is always in my list of forests
09:06 < adiabat> one variation which I had but took out a while (but should try again) is "SibSwap", which swaps siblings when both are known and left > right
09:06 < adiabat> ah OK yes that makes sense
09:06 < adiabat> there are many other algorithms that can work with the incoming data we have
09:07 < adiabat> and some of them are likely much better in terms of proof sizes
09:07 < adiabat> well, probably not "much"
09:07 < ja> but i want to make it equal to remtrans, i am just having a hard time figuring out how remtrans actually works, since my algorithm is totally different
09:07 < adiabat> I don't know though, if there's actual different algorithms that are significantly smaller / faster it would certainly be worth switching
09:08 < ja> so many i should actually just plot all those different options
09:08 < ja> but how to develop a sensible benchmark?
09:09 < adiabat> yeah that's the hard part, the benchmark would really be against observed mainnet data
09:09 < adiabat> since it's using the heuristics of new UTXOs are often spent quicky, etc
09:09 < ja> my algorithm is just copying over perfect subtrees with no proofs until there are no more left, and then the remaining leaves can be copied over. but as shown in my example above, if i just take those leaves in order, it won't work, they need swizzling
09:10 < adiabat> though it's modular enough now that if you replace remtrans2 with some other variat that works, then do a mainnet IBD, it should work
09:11 < adiabat> yeah for example right now in remtrans, the first thing it does is extractTwins
09:11 < adiabat> which skips over all "twin" deletions
09:11 < adiabat> but you could move things into those spots in that they're all proven and available for hashes to swap into
09:12 < adiabat> skipping over twins (& flagging the parent) is the simplest way to go, but there's lots of other things you could do
09:13 < ja> but why only twins? why not quads? the point is just to find large perfect subtrees, no?
09:14 < adiabat> well if you run extractTwins it'll also get rid of runs of 4, 6, 8 etc
09:14 < adiabat> and then the latter functions rely on the fact that all deletions are next to a non-deleted but known hash
09:15 < adiabat> hm actually maybe it would still work if you just took out extractTwins()
09:15 < adiabat> in makeSwaps the main line is 
09:15 < adiabat> rowSwaps[i] = arrow{from: dels[1] ^ 1, to: dels[0]}
09:16 < adiabat> if dels[0] is the sibling of dels[0] it will just swap them, which does nothing
09:16 < adiabat> er
09:16 < adiabat> dels[0] the sibling of dels[1], which means dels[1]^1 == dels[0]
09:16 < adiabat> so you get an arrow{ dels[0], dels[0] }
09:17 < adiabat> so maybe you can just comment out extractTwins and it still works, just with a bunch of swaps that do nothing
09:22 < ja> hmm, so that would mean dels for makeSwaps would always be empty?
09:23 < ja> ah no, it would just remain the same
09:25 < ja> so when you say it would have a bunch of swaps that do nothing, you mean, the forest would be equivalent? or merely that it would still conform to the swapping rules (can't swap rehash nodes with no proof for them)?
09:30 < adiabat> I think it would end up with the same forest
09:31 < adiabat> so just removing extractTwins() alone might not change anything
09:31 < adiabat> but if you remove extractTwins(), you could change makeSwaps() 
09:32 < adiabat> for example, making it so that the first non-twin deletion sibling moves to the first deletion
09:32 < adiabat> even if that first deletion is a twin
09:33 < adiabat> that would result in a different forest, but would still work in that it would all be hashable
09:37 < ja> interesting, i will experiement
10:16 < adiabat> sure let me know if there's anything I can explain better.  it is kindof confusing... still not the best docs
--- Log closed Sat Jan 09 00:00:10 2021