--- Log opened Fri Mar 05 00:00:47 2021
08:33 < adiabat_> yeah the log scale color plots are cool
08:34 < adiabat_> can see a lot more...
08:34 < adiabat_> I would maybe change the yellow to red or something, as it blends in with the white background so it's a bit hard to see the yellow edge
08:35 < adiabat_> this also shows some really weird things
08:35 < adiabat_> the vertical bands that fade as they go downwards... what could cause that?
08:36 < adiabat_> the 600K image shows them really well, but they're in every image
08:37 < adiabat_> there's a clear one around leaf position 6M, around height 648K or so
08:38 < adiabat_> but yeah they're all over the place.  A bright line in a region, and then it fades back down to background noise over thousands of blocks
08:38 < adiabat_> I have a guess as to what these are: high row swaps
08:39 < kcalvinalvin> Yeah the colors are sorta hard :(. You have to pick from one of these https://matplotlib.org/stable/tutorials/colors/colormaps.html. I can re-do the plot to the color you think is good
08:39 < adiabat_> where a swap happens up at height 15 or so, moving a bunch of new leaves from the yellow region deep into the blue region
08:40 < adiabat_> and those UTXOs are new, and so they keep getting spent at a high rate, so you get the band as they get spent / die off
08:42 < adiabat_> also like around height 633K, there's a bunch of really thin bands that show up in the 1.6 - 2.2M leaf region
08:42 < adiabat_> wait no that's 16M to 22M
08:42 < adiabat_> I guess that happens when someone spends really old UTXOs.  It creates a gap way back to the left, and something's gotta fill that gap
08:43 < adiabat_> and whatever you try to fill it with, it's going to be "hotter" than the old stuff around it
08:43 < adiabat_> and then that keeps happening; 300K block old UTXO get spent, makes a gap, fill in the gap with a new utxo
08:44 < adiabat_> new utxo gets spent 3 blocks later, same gap opens up, fill it in with a new utxo
08:44 < adiabat_> repeat until eventually someone puts a utxo there that they will sit on (hodl)
08:44 < kcalvinalvin> Also one other thing that I noticed. Some of the horizontal greens match up with the utxo spending pattern visualized here https://utxo.live/triangle/. The problem with utreexo stuff is that almost all of the forest gets changed (instead of maybe to like the middle)
08:45 < kcalvinalvin> oooh hmm yeah the vertical stuff makes sense
08:46 < adiabat_> yeah there are bands like at height 510K, 542-549K, etc
08:46 < kcalvinalvin> I guess in that sense the locality isn't so bad for cowforest caching? Since the same range will get accessed
08:46 < adiabat_> those are times when we know there was a lot of consolidation
08:47 < kcalvinalvin> For proofs I think they hurt
08:47 < adiabat_> right, there doesn't seem to be any way around that though
08:47 < adiabat_> people spending old stuff is going to be harder than spending new stuff
08:48 < kcalvinalvin> If people hodl more we can ibd faster
08:48 < kcalvinalvin> HODL
08:48 < adiabat_> these are super cool charts though, should sell them to like blockchain analytics traders or something
08:48 < adiabat_> yeah the vertical bands do seem... sub optimal
08:48 < adiabat_> as they're persistent and way in the middle...
08:49 < adiabat_> also it doesn't always happen.  the large horizontal bands don't really fade out, they just cut off
08:49 < kcalvinalvin> those empty vertical bands are great
08:49 < adiabat_> empty?  oh like in the ancient blocks?
08:50 < kcalvinalvin> like up til block 300,000 you have that empty veritcal band
08:50 < adiabat_> it just looks bigger because the scale is 100X smaller in the old charts
08:50 < kcalvinalvin> ahhh
08:50 < adiabat_> someone made a couple thousand spam txs back before op_return was a thing I guess
08:50 < adiabat_> and those utxos are still there forever...
08:51 < adiabat_> probably not worth it but we could see what those are and if they're unspendable could alter the isUnspendable() call to ignore them
08:51 < kcalvinalvin> What if they're spendable
08:51 < adiabat_> well then have to leave them :)
08:51 < adiabat_> very likely lost by now
08:52 < kcalvinalvin> Overall, I guess the locality isn't bad
08:52 < kcalvinalvin> Some stuff to fix
08:52 < adiabat_> first 100K blocks is easy, not much going on
08:52 < adiabat_> yeah there are things that are not perfect but... I don't know how you'd fix them
08:52 < kcalvinalvin> first 100k blocks take like 10 seconds to verify heh
08:53 < adiabat_> vertical banding does seem like maybe there some way to improve... hmmm
08:53 < adiabat_> swapSibs might help?
08:53 < kcalvinalvin> We should prioritize stuff. Hashtoknown first, swapSibs second
08:53 < adiabat_> yeah
08:53 < kcalvinalvin> And I get the parallel stuff working
08:54 < kcalvinalvin> Undo... is and odd one
08:54 < adiabat_> undo is annoying
08:54 < kcalvinalvin> How would undo work into the current utreexoserver stuff
08:54 < kcalvinalvin> I can see how it'd fit in for utcd
08:54 < kcalvinalvin> But... utreexoserver?
08:55 < adiabat_> yeah it doesn't, the server can't reorg
08:55 < adiabat_> it doesn't see new stuff come in anyway so
08:56 < adiabat_> we could manually tell it to reorg
08:56 < kcalvinalvin> For testing I can see how we'd fit it in
08:57 < adiabat_> yeah a bit part of undo is getting the right testing setup for it
08:57 < adiabat_> because previously I'd like, get it to work, and have some little test, but then nothing ever actually used it
08:57 < adiabat_> so then it would stop working and wouldn't realize
09:01 < kcalvinalvin> Some extensive bitcoin regtest would have to do
09:01 < adiabat_> hmmm. looking at these histograms more
09:01 < adiabat_> maybe TTL is not even worth it?
09:01 < kcalvinalvin> And also test with testnet3
09:01 < adiabat_> just take the right edge, that's pretty much everything
09:01 < kcalvinalvin> what do you mean
09:01 < adiabat_> I guess TTL values will tell you about the vertical bands so you'll cache those regions
09:02 < adiabat_> like the point of TTL is to tell you where the yellow stuff will be
09:02 < adiabat_> but.. it's always on the right edge, so you don't need it to tell you anything
09:02 < adiabat_> I guess TTL / clairvoyant helps with the vertical bands though
09:03 < adiabat_> if we could get rid of all the vertical bands, then TTL data wouldn't be needed
09:03 < adiabat_> but I don't think you ever really can.  but yeah to the extent that the histogram is a totally smooth yellow edge, you don't need TTL / any fancy caching
09:04 < kcalvinalvin> That'd simplify the code
09:06 < kcalvinalvin> hmmmm. So does that mean you got the clairvoyant stuff right the first time?
10:22 < adiabat_> probably not, but it might be fairly close either way
--- Log closed Sat Mar 06 00:00:48 2021