--- Day changed Wed Jan 29 2020
02:32 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has quit [Remote host closed the connection]     
03:05 -!- belcher [~belcher@unaffiliated/belcher] has joined #bitcoin-core-pr-reviews     
03:05 -!- Gerardo23Little [~Gerardo23@ns334669.ip-5-196-64.eu] has joined #bitcoin-core-pr-reviews     
03:11 -!- Gerardo23Little [~Gerardo23@ns334669.ip-5-196-64.eu] has quit [Ping timeout: 268 seconds]     
03:38 -!- orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
03:42 -!- pk1 [~pk@158.140.254.215] has quit [Quit: WeeChat 2.7]     
03:48 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
03:49 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Client Quit]     
04:22 < provoostenator> Pre-rant (since I can't make it tonight): I hate how IsSpent(), which calls out.IsNull(), can mean both "unspent" and "doesn't exist".
04:23 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has joined #bitcoin-core-pr-reviews     
05:37 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has quit [Ping timeout: 248 seconds]     
06:02 -!- setpill [~setpill@unaffiliated/setpill] has joined #bitcoin-core-pr-reviews     
06:20 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has quit [Remote host closed the connection]     
06:20 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has joined #bitcoin-core-pr-reviews     
06:48 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has joined #bitcoin-core-pr-reviews     
07:04 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has quit [Ping timeout: 248 seconds]     
07:05 -!- csknk [~csknk@unaffiliated/csknk] has joined #bitcoin-core-pr-reviews     
07:16 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
07:20 -!- molz_ [~molly@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
07:23 -!- molly [~molly@unaffiliated/molly] has quit [Ping timeout: 265 seconds]     
07:37 -!- molz_ [~molly@unaffiliated/molly] has quit [Ping timeout: 272 seconds]     
07:43 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
07:45 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
07:53 -!- felixfoertsch [~felixfoer@92.117.56.158] has quit [Read error: Connection reset by peer]     
07:53 -!- felixfoertsch23 [~felixfoer@92.117.56.158] has joined #bitcoin-core-pr-reviews     
08:00 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
08:02 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
08:06 -!- jamesob [sid180710@gateway/web/irccloud.com/x-nezxkhuukhlgorqy] has joined #bitcoin-core-pr-reviews     
08:15 -!- setpill [~setpill@unaffiliated/setpill] has quit [Quit: o/]     
08:16 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
08:19 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
08:22 < jnewbery> provoostenator: PR to add comment to https://github.com/bitcoin/bitcoin/blob/c1607b5df4877e5f799d861784cb91dba3ea5887/src/coins.h#L76 welcome :)
08:27 -!- Talkless [~Talkless@hst-227-49.splius.lt] has joined #bitcoin-core-pr-reviews     
08:27 -!- Talkless [~Talkless@hst-227-49.splius.lt] has quit [Client Quit]     
08:28 -!- Talkless [~Talkless@hst-227-49.splius.lt] has joined #bitcoin-core-pr-reviews     
08:34 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
08:37 -!- davterra [~dulyNoded@172.98.86.80] has joined #bitcoin-core-pr-reviews     
08:48 -!- dr-orlovsky [~dr-orlovs@194.230.147.122] has joined #bitcoin-core-pr-reviews     
08:54 -!- dr-orlovsky [~dr-orlovs@194.230.147.122] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
09:08 -!- jonatack [~jon@54.76.13.109.rev.sfr.net] has joined #bitcoin-core-pr-reviews     
09:12 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Ping timeout: 240 seconds]     
09:12 -!- jonatack [~jon@54.76.13.109.rev.sfr.net] has quit [Ping timeout: 265 seconds]     
09:12 -!- ghost43_ [~daer@gateway/tor-sasl/daer] has joined #bitcoin-core-pr-reviews     
09:13 -!- jonatack [~jon@213.152.161.170] has joined #bitcoin-core-pr-reviews     
09:14 -!- emilengler [~emilengle@unaffiliated/emilengler] has joined #bitcoin-core-pr-reviews     
09:20 -!- andrewtoth [~andrewtot@gateway/tor-sasl/andrewtoth] has joined #bitcoin-core-pr-reviews     
09:23 -!- SirRichard [~MaxSikors@cpe-98-28-69-149.columbus.res.rr.com] has joined #bitcoin-core-pr-reviews     
09:23 < SirRichard> hi     
09:24 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
09:38 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Ping timeout: 265 seconds]     
09:42 -!- lightlike [~lightlike@p200300C7EF0F5F0088CA903BA173634E.dip0.t-ipconnect.de] has joined #bitcoin-core-pr-reviews     
09:45 -!- hanhua [68850952@104.133.9.82] has joined #bitcoin-core-pr-reviews     
09:47 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
09:52 -!- pelt [~pelt@195.181.160.175.adsl.inet-telecom.org] has joined #bitcoin-core-pr-reviews     
09:54 -!- jungly_ [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
09:54 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Read error: Connection reset by peer]     
09:58 < jamesob> Okay folks, in a few minutes we're going to get started on the meeting for https://bitcoincore.reviews/17487.html
09:59 -!- pglazman [~pglazman@205.209.24.227] has joined #bitcoin-core-pr-reviews     
10:00 < jamesob> #startmeeting
10:00 < fjahr> hi     
10:00 < jamesob> hi     
10:00 < ajonas> hi     
10:00 < jonatack> hi     
10:00 < jamesob> "We usually start Bitcoin Core IRC meetings with a 'hi' so it's clear who's at keyboard. Feel free to say hi, even if you arrive in the middle of the meeting!" in the words of jonatak
10:00 < emzy> hi     
10:00 < andrewtoth> hi     
10:00 < jnewbery> hi     
10:01 < jamesob> quick reminder about meeting conventions:
10:01 < jonatack> jamesob: that would be jnewbery :)
10:01 < jamesob> You don't have to ask to ask a question (e.g. "I have a question about x but I don't know if it's on-topic?"). Just go ahead and ask. If it's off-topic we'll tell you.
10:01 < jamesob> I'm here to help moderate, not to lead. You don't need to wait for me to ask a specific question — just jump in at any point.
10:01 < nehan_> hi     
10:01 < jamesob> (also in the words of one of the other Js)
10:01 < kanzure> hi     
10:02 -!- billygarrison [c6a6d605@iusr5.gov.ns.ca] has joined #bitcoin-core-pr-reviews     
10:02 < jamesob> okay, so today we're going to be covering https://github.com/bitcoin/bitcoin/pull/17487
10:02 < jamesob> Who had a chance to read the PR? (y/n)
10:02 < fjahr> y     
10:02 < andrewtoth> y     
10:02 < jonatack> y     
10:02 < nehan_> .5*y
10:02 < jamesob> okay we can do floats too
10:02 < lightlike> hi     
10:03 < jnewbery> ~.5     
10:03 < emzy> y     
10:03 < jkczyz> .5 hi
10:03 < jamesob> In summary, the PR allows writing the contents of the UTXO set down to disk without wiping out the in-memory cache, which has some performance benefits.
10:03 < jamesob> The parameter that allows this isn't actually used in the PR.
10:04 < jamesob> Who had a chance to review the PR? (y/n)
10:04 -!- michaelfolkson [~textual@host-92-6-97-247.as43234.net] has joined #bitcoin-core-pr-reviews     
10:05 < fjahr> wip     
10:05 < emzy> n     
10:05 < andrewtoth> 0.5 y
10:05 < jamesob> (well, the parameter isn't actually used aside from accompanying tests)
10:05 < jonatack> first commit
10:05 < jonatack> + concept
10:05 < lightlike> also only 1st commit
10:06 < jamesob> that's great. Before we start talking about things like cache flushing, though, and in-memory vs. on disk we should probably get some things straight conceptually
10:06 < jamesob> Can anyone tell me what the "shape" of the UTXO set is? I.e. what is it keyed and valued by?
10:06 < fjahr> key: COutPoint vlaue: Coin
10:07 < pelt> hi     
10:07 < jamesob> correct. what's an outpoint?
10:07 < andrewtoth> key: txid:vout value: scriptpubkey,amount,height in block
10:08 < fjahr> txid and output pos
10:08 < jonatack> the data structure used to refer to a transaction output
10:08 < michaelfolkson> "The data structure used to refer to a particular transaction output, consisting of a 32-byte TXID and a 4-byte output index number (vout)."
10:08 < jamesob> andrewtoth: that's mostly correct, but you're missing a few important things in terms of the UTXO set data structure
10:09 < jonatack> i cribbed that from michaelfolkson
10:09 < jamesob> instead of being valued by a Coin, which has the constituent data you've listed, it's actually valued by a CCoinsCacheEntry, which has some extra stuff
10:09 < jamesob> anyone know what that extra stuff is?
10:09 < jnewbery> flags
10:09 < andrewtoth> it appears to just be flags for DIRTY and FRESH and then the Coin
10:09 < nehan_> flags
10:10 < jamesob> jnewbery andrewtoth nehan_: bingo. we'll come back to those in a bit
10:10 < jamesob> this wasn't *always* the structure of the UTXO set. can anyone tell me how it looked previously?
10:11 < jamesob> long story short, it used to be keyed by txid and valued by an array of coins. more info in this PR: https://github.com/bitcoin/bitcoin/pull/10195
10:12 < jamesob> (but that's an aside)
10:12 < jamesob> okay, so now let's talk about the different layers of the UTXO set's implementation
10:12 < jnewbery> hint: it changed in 0.15
10:13 < jamesob> there are a few different classes to be aware of here. Who can tell me what the CCoinsView class is?
10:13 < andrewtoth> ahh yes i remember, there was a dos vector there where it would have to pull in lots of coins into memory doing it that way
10:13 < jamesob> andrewtoth right, and I think it was in general less efficient for access
10:14 < jamesob> hint: https://github.com/bitcoin/bitcoin/blob/master/src/coins.h#L153-L155
10:14 < lightlike> did every node that upgraded to 0.15 have to reindex-chainstate back then?
10:14 < fjahr> It basically represents the whole UTXO set
10:14 < jamesob> I believe so - there was a migration function written somewhere
10:15 < jamesob> fjahr: in a sense. CCoinsView defines the interface of the UTXO set, independent of any given storage medium
10:15 < nehan_> CCoinsView is the common interface for all the different layers
10:15 < jamesob> nehan_: right!
10:15 < jonatack> A txout dataset abstraction
10:15 < jnewbery> upgrade code here: https://github.com/bitcoin/bitcoin/blob/c1607b5df4877e5f799d861784cb91dba3ea5887/src/txdb.cpp#L352
10:15 < jamesob> so there isn't an actual instantiation of a CCoisnView anywhere, but there are a few concrete subclasses
10:15 < nehan_> I don't really understand exactly *why* there have to be so many layers, but maybe we will get to that
10:16 < jamesob> nehan_: yep, we'll get there pretty shortly
10:16 < jamesob> can anyone tell us what CCoinsViewDB does?
10:16 < jamesob> (which is a subclass of CCoinsView, unsurprisingly)
10:17 < jamesob> (https://github.com/bitcoin/bitcoin/blob/master/src/txdb.h#L42-L44)
10:17 < andrewtoth> it's an implementation of the interface to represent the state of the utxo set in leveldb
10:17 < jamesob> andrewtoth: right!
10:17 < nehan_> manages the leveldb
10:18 < jamesob> you can think of this in very coarse terms as being the conduit to storing and reading UTXOs to and from disk
10:19 < jamesob> so there's another class called CCoinsViewBacked, can anyone tell us what that does? it's a little abstract
10:19 < jamesob> (https://github.com/bitcoin/bitcoin/blob/master/src/coins.h#L190-L192)
10:20 -!- nadra [uid415365@gateway/web/irccloud.com/x-fjxllcozpkddcykh] has joined #bitcoin-core-pr-reviews     
10:20 < nehan_> I guess it's purpose is so that you can change base?
10:21 < nehan_> it's just a 1:1 wrapper around whatever is backing it, right?
10:21 < fjahr> backed by something else than the db I guess
10:21 < jamesob> nehan_: yep, basically
10:21 < jamesob> it's a CCoinsView implementation that says "I'm one layer of a cache, but there's another coins view that sits behind me, and I'll consult that view if the user of me asks for a coin I don't have."
10:22 < jamesob> so there's a subclass of CCoinsViewBacked called CCoinsViewCache: surely someone can tell us what this does
10:22 < jonatack> It seems to be used in case of shutdown
10:22 < jonatack> in CCoinsViewErrorCatcher
10:23 < jamesob> jonatack: yep, that's an interesting but sort of tangential use of CCoinsViewBacked
10:23 < jonatack> src/coins.h#L338-359
10:23 < nehan_> jamesob: Couldn't all the sub classes have just overriden CCoinsView directly? Why have an intermediate CCoinsViewBacked?
10:23 < jamesob> CCoinsViewErrorCatcher exists basically so that we can propagate leveldb errors to sensible error messages in the GUI
10:23 < andrewtoth> it maintains an in memory cache of the utxo set, *backed* by the leveldb representation of the utxo set
10:24 < lightlike> CCoinsViewCache seems to be the actual in-memory cache of limited size that is backed by the db
10:24 < jamesob> nehan_: good question. It turns out (I think) that is because CCoinsViewDB doesn't have a cache behind it, and so it subclasses CCoinsView directly
10:24 < jonatack> oh right, src/coins.h#L209
10:24 < jamesob> it's the only view that does so
10:24 < jamesob> andrewtoth lightlike: yep, exactly right
10:25 < jamesob> so! to get around to nehan_'s earlier question, we arrange different instances of these caches to relate to one another
10:25 < jamesob> can someone tell us what this "stack" of coins views looks like?
10:26 < nehan_> this was very helpful: https://jameso.be/dev++2018/#54
10:26 < jnewbery> nehan_: +1
10:26 < jonatack> agreed
10:27 < jamesob> thanks for linking :)
10:28 < jamesob> so basically, the arrangement looks like this:
10:28 < jamesob> CCoinsViewDB (disk; i.e. canonical store) <- CCoinsViewCatcher (leveldb errors to GUI) <- CCoinsViewCache (in-memory cache)
10:28 < emzy> nehan_: +1
10:29 < jamesob> so on the top of the cache is the in-memory store. If we try to fetch a coin from that view and it fails, it'll try the fetch from CCoinsViewCatcher. CCoinsViewCatcher doesn't store *anything*, so it will always fall back to CCoinsViewDB, which ultimately uses a class called DBWrapper to consult leveldb (which is the on-disk local database we use)
10:30 < jamesob> so as you can see this is pretty complicated.
10:30 < jamesob> why don't we just do everything in memory?
10:30 < lightlike> will said coin be automatically added to the in-memory cache in this case?
10:31 < jamesob> lightlike that's a good question and we're jumping ahead a little bit, but yes: CCoinsViewCache pulls any fetched coin into its cache after it fetches it from its parent cache
10:31 < nehan_> jamesob: "on the top of the cache is the in-memory store" --> do you mean on top of CCCoinsViewCatcher? Is there another in-memory store on top of CCoinsViewCache?
10:32 < jonatack> s/CCoinsViewCatcher/CCoinsViewErrorCatcher/ ??
10:32 < jamesob> sorry that was poorly phrased on my part - what I mean is that the CCoinsViewCache is the top layer of the cache
10:32 < jamesob> jonatack: right, thanks
10:32 < jamesob> nehan_: your second question is a great one
10:32 < nehan_> jamesob: ah, got it thanks
10:32 < jamesob> yes, sometimes we stack CCoinsViewCaches on top of one another
10:33 < lightlike> not automatically added because of memory limitations - the utxo set is too large for many computers.
10:33 < jonatack> jamesob: thanks, makes more sense to me now :D
10:33 < jamesob> Sometimes in-memory caches (CCoinsViewCache instances) sit on top of one another, e.g. when connecting a block (https://github.com/jamesob/bitcoin/blob/2ed74a43a05a47129d56117deeb489addbcaf05f/src/validation.cpp#L2586-L2599) to enforce atomicity.
10:33 < michaelfolkson> jamesob: Memory restrictions will cause crashes
10:33 < jamesob> lightlike michaelfolkson: correctamoondo
10:34 < jamesob> and beyond that, there are durability considerations: if we crash while everything is in memory, the UTXO set gets wipe out and we'd have to reindex on startup
10:35 < jamesob> so then why don't we just use leveldb for everything, skip the memory stuff?
10:35 < michaelfolkson> Performance, longer sync time?
10:35 < emzy> just a thought, it is to slow.
10:35 < nehan_> jamesob: this seems like a really weird way to use what should be just an in-memory cache. it's sort of being used as a way of atomically applying updates?
10:36 < jamesob> yup. Memory is way slower than disk as many of you probably know (https://stackoverflow.com/questions/1371400/how-much-faster-is-the-memory-usually-than-the-disk)
10:36 < jamesob> oops, vice versa
10:36 < jamesob> disk is way slower than memory
10:36 < jonatack> whew
10:36 < nehan_> i'm surprised leveldb doesn't do its own caching
10:36 < jamesob> nehan_: good point. it does to a certain extent, but it isn't sophisticated and we set it to be pretty limited
10:37 < jamesob> that'd be worth looking into
10:38 < jamesob> nehan_: yeah, in the case I link to, we use the temporary CCoinsViewCache as a sort of "transaction" (in the database sense) to be able to easily "roll back" if any of the spends don't validate when connecting a block
10:38 < jnewbery> jamesob: in that example you used of a cache on top of another, this final call to flush(): https://github.com/jamesob/bitcoin/blob/2ed74a43a05a47129d56117deeb489addbcaf05f/src/validation.cpp#L2597 is what causes the UTXO set to be updated in the case that the block is successfully connected? If block connection fails then we can just throw away `view` and the underlying UTXO set doesn't
10:38 < jnewbery> get updated?
10:39 < jamesob> exactly - the changes don't propagate to our "actual" UTXO set until we call `Flush()` there
10:39 -!- jungly_ [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Remote host closed the connection]     
10:39 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
10:40 < jamesob> so when to Flush() actually turns out to be pretty important because as you can probably guess, it not only amounts to writing to disk but (currently) it removes everything from the in-memory portion of the view structure
10:40 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Remote host closed the connection]     
10:40 < jamesob> this PR, of course, makes the latter part of that optional
10:40 < jamesob> but we're still not quite done talking about the CoinsView internals. let's return to those flags we were talking about on CCoinsCacheEntry
10:40 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
10:41 < jamesob> in order to make efficient use of the in-memory cache, we want to be able to avoid writing to and reading from disk if, say, a coin is created and spent without running low on memory
10:42 < jamesob> (when we run low on memory, we `Flush()`)
10:42 < jamesob> those flags are key in helping us do that
10:42 < jamesob> so what does the FRESH flag indicate when it is true for a cache entry?
10:42 < jonatack> src/coins.h#L120 FRESH is a performance optimization with which we can erase coins that are fully spent if we know we do not need to flush the changes to the parent cache.  It is always safe to not mark FRESH if that condition is not guaranteed.
10:43 < nehan_> I found that comments confusing. what is the "condition" exactly?
10:43 < nehan_> s/comments/comment
10:43 < jamesob> jonatack: right. FRESH basically says "the cache that sits behind me has never seen this entry - I haven't flushed it yet. So if the entry gets removed, I can just remove it from my cache - no need to tell my parent."
10:43 < jamesob> nehan_: I think the "condition" is "my parent hasn't seen this entry"
10:43 < andrewtoth> this is basically an optimization to avoid writing a transient utxo. if the db hasn't seen it and it is spent, it never needs to be written
10:44 < jamesob> so there are certain cases where you may not know if leveldb has seen the entry or not, i.e. when we are recovering from a crash that happened during a Flush()
10:44 < pelt> If the system crashes while flush() is running, it would require the node to reindex.  Is that correct?
10:44 < jonatack> i read it as generally, when in doubt, don't mark as FRESH but it's nice if we can
10:44 < jamesob> pelt: luckily that isn't the case - there is a mechanism built into CCoinsViewDB that allows us to recover from such a crash without reindex
10:45 < nehan_> in my experience one usually uses the term "dirty" to indicate that an entry is different in the cache than in its underlying store
10:45 < jnewbery> nehan_: that's what we're using here
10:45 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Ping timeout: 265 seconds]     
10:45 < jnewbery> https://github.com/bitcoin/bitcoin/blob/c1607b5df4877e5f799d861784cb91dba3ea5887/src/coins.h#L118-L124
10:45 < nehan_> ok. fresh means was dirty but not anymore?
10:46 < pelt> jamesob soming to the effect of, since we can tell node when we're done even if we crash on flush() disregard the delta on CCoinsViewDB?
10:46 < andrewtoth> nehan_ fresh means dirty and not seen in db
10:46 < jnewbery> Imagine we receive a block that has transaction A and transaction B, which spends one of A's outputs. We apply that block atomically and never need to apply that output to our UTXO set when we flush
10:46 < jamesob> nehan_: fresh implies dirty, I think
10:46 < nehan_> o_O     
10:47 < jnewbery> because we've marked A's output as FRESH, when we spend it, we can just remove it from our cache instead of flushing it at the end of block connection
10:47 < jamesob> jnewbery: right!
10:47 < fjahr> hm, can something be dirty if there is no underlying store yet? never thought of that...
10:47 < nehan_> jnewbery: thanks
10:47 < jnewbery> I think it's worth pointing out that the comment is a bit out of date (the words 'pruned' and 'fully spent' only made sense when the UTXO was tracking transactions instead of outputs
10:48 < jamesob> DIRTY basically means "this entry somehow differs from the entry my parent would have"
10:48 < lightlike> i understand the use case for fresh, but not yet the one for DIRTY.
10:48 < jnewbery> just ignore the words 'pruned' and 'fully'
10:49 < jamesob> jnewbery: I initially thought that, but "pruned" does still have a relevant meaning: because parent caches can return null values for coins which have been spent in their caches, those null values are sometimes referred to as "pruned"
10:49 < jamesob> anyway, suffice to say: this stuff can get complex!
10:49 < jamesob> the way that fresh and dirty flags work, when they are set, and why is potentially the hardest thing to understand about the innerworkings of the UTXO view structure
10:50 < andrewtoth> if the utxo is brand new, it is DIRTY because the db doesn't have it, so it's different. It's also FRESH, since the db doesn't have it. If it gets spent before a write, it can just be removed
10:50 < jnewbery> huh, I'll need to look into that. I thought 'pruned' meant 'all the outputs of this tx have been spent, we can remove it from our (pre-0.15 per-transaction) UTXO set. No need to discuss here though!
10:50 < jamesob> but they allow us to fully leverage the in-memory cache by pruning coins which are created and spent in between disk flushes without ever touching the disk
10:51 < jamesob> andrewtoth: right
10:51 < nehan_> andrewtoth: thank you! that's a good explanation.
10:51 < jamesob> but a coin that has already been written to the parent cache but is then spent should be marked DIRTY
10:51 < nehan_> so the writer needs to be confident that this coin is absolutely not in the parent cache.
10:51 < nehan_> when marking things fresh
10:52 < jamesob> yep, right
10:52 < jamesob> so - the bulk of this pull request is basically this chunk of code: https://github.com/bitcoin/bitcoin/pull/17487/files#diff-cd7b305fd4b4280f22ae88960e60398eR212-R227
10:52 < jamesob> can someone describe it in high level terms?
10:53 < jamesob> (we're going to more or less stop in 5 minutes in case you all have any burning questions)
10:55 < jnewbery> It's a flush, but without emptying the cache. You still need to clear the flags because they're invalidated by writing to the layer below
10:55 < jonatack> jamesob: fwiw i wondered if this first commit wouldn't be better separated into 2 commits, one that adds Sync() and one the adds bool erase
10:55 < jamesob> jnewbery: right! thanks
10:55 < jonatack> that*
10:56 < jamesob> jonatack: yeah could be - would need to look at it again, but it may be a small enough change that that'd be a stylistic call
10:56 -!- billygarrison [c6a6d605@iusr5.gov.ns.ca] has quit [Remote host closed the connection]     
10:56 < jamesob> anyway, early in the PR there was a consensus bug based upon not resetting those flags when flushing without erase
10:56 < jamesob> I'm curious if anyone can tell us how that bug works
10:57 < jamesob> (and/or chime in with whatever questions you'd like because we're almost out of time)
10:57 < nehan_> i think it's leaving a coin marked as "fresh" even though it has been sync'd to the parent
10:58 < emzy> Any practical numbers of the speedup?
10:58 < nehan_> so you might never sync the spend of the coin
10:58 < nehan_> and i suppose you could spend a utxo again
10:58 < jonatack> the logical danger would be setting FRESH when it shouldn't be
10:58 < lightlike> could it be interesting idea for the future to try partial flushs (according to some heuristic) so that we free up some space but hopefully keep those coins in memory that we may need soon?
10:58 < jamesob> nehan_: right!
10:58 < nehan_> good catch by sjors
10:58 < jamesob> emzy: see https://github.com/bitcoin/bitcoin/pull/17487#issuecomment-557595582 and subsequent bench results
10:59 < jamesob> yeah, all due props to sjors
10:59 < michaelfolkson> Any good resources for better understanding Core's use of LevelDB and databases? This looks promising https://bitcoindev.network/understanding-the-data/
10:59 < jamesob> lightlike: yep, I have a branch that tried this but surprisingly I didn't get good results
11:00 < jonatack> yes, bien joué monsieur sjors * claps
11:00 < jamesob> michaelfolkson: there aren't a ton of resources on that surprisingly, but I can recommend a very broad talk I did back in 2018: https://jameso.be/dev++2018/#1
11:00 -!- billygarrison [c6a6d605@iusr5.gov.ns.ca] has joined #bitcoin-core-pr-reviews     
11:00 < jamesob> there's a video for that somewhere too
11:00 < jnewbery> michaelfolkson: I hadn't seen that before, but from a very quick skim it looks decent
11:01 < jamesob> but in general, best thing to do is review the database/coins code in coins.{h,cpp}, txdb.{h,cpp}, and FlushStateToDisk in validation.cpp
11:01 < michaelfolkson> jamesob: Cool thanks, I will watch that.
11:01 < jamesob> okay folks, that's all the time we have
11:01 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has quit [Remote host closed the connection]     
11:01 < jamesob> thanks for showing up!
11:01 < jamesob> #endmeeting
11:01 < fjahr> thanks jamesob
11:01 < nehan_> thanks!
11:01 < andrewtoth> thanks jamesob! and thanks to all organizers
11:01 < jnewbery> That was great. Thanks James!
11:01 < lightlike> thanks! learned a lot today!
11:01 < emzy> tnx jamesob
11:02 < billygarrison> Thanks James and participants! I learned a lot as a fly on the wall.
11:02 < jonatack> michaelfolkson: interesting. that link was by grokchain, who was present at review club lately
11:02 < michaelfolkson> Thanks for your patience jamesob! I'll get there one day haha
11:02 < ajonas> Talk that nehan_ and James referenced is here for those interested: https://www.youtube.com/watch?v=L_sI_tXmy2U
11:02 < jamesob> michaelfolkson: keep at it!
11:02 < jnewbery> You're welcome to come back and do another assumeUTXO PR whenever you want!
11:02 < jonatack> jamesob: awesome meeting and notes. learned tons.
11:03 < jonatack> thanks!
11:03 < billygarrison> jamesob excellent intro written on the review page btw - gave great context for a newbie like me!
11:03 < jamesob> jnewbery: careful what you wish for!
11:03 < pelt> thanks all
11:03 < jamesob> billygarrison: glad to hear it
11:03 < jamesob> jonatack jnewbery: thanks for putting this together. great idea!
11:04 < jonatack> +1 assume-utxo follow-up
11:04 < pelt> was very helpful to better understand CCoinsView* layering
11:05 < jamesob> pelt: glad to hear. it's pretty complicated
11:07 < jnewbery> here's a write-up of the UTXO changes in v0.15: https://johnnewbery.com/post/whats-new-in-bitcoin-core-v0.15-pt1/
11:07 < jnewbery> (I didn't want to derail the conversation during the meeting)
11:08 < michaelfolkson> Thanks
11:11 -!- billygarrison [c6a6d605@iusr5.gov.ns.ca] has quit [Remote host closed the connection]     
11:12 -!- lightlike [~lightlike@p200300C7EF0F5F0088CA903BA173634E.dip0.t-ipconnect.de] has quit [Quit: Leaving]     
11:16 -!- michaelfolkson [~textual@host-92-6-97-247.as43234.net] has quit [Quit: Sleep mode]     
11:40 -!- vasild_ [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews     
11:43 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]     
11:53 -!- vasild_ is now known as vasild     
12:03 -!- pglazman [~pglazman@205.209.24.227] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]     
12:10 -!- dr-orlovsky [~dr-orlovs@194.230.147.122] has joined #bitcoin-core-pr-reviews     
12:12 -!- pelt [~pelt@195.181.160.175.adsl.inet-telecom.org] has quit [Quit: pelt]     
12:14 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has joined #bitcoin-core-pr-reviews     
12:14 -!- orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
12:17 -!- dr-orlovsky [~dr-orlovs@194.230.147.122] has quit [Ping timeout: 265 seconds]     
12:29 -!- jungly_ [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
12:30 -!- emilengler [~emilengle@unaffiliated/emilengler] has quit [Quit: Leaving]     
12:31 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
12:31 -!- jungly_ [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Read error: Connection reset by peer]     
12:33 -!- Talkless [~Talkless@hst-227-49.splius.lt] has quit [Quit: Konversation terminated!]     
12:48 -!- SirRichard [~MaxSikors@cpe-98-28-69-149.columbus.res.rr.com] has quit [Quit: SirRichard]     
12:57 -!- jonatack [~jon@213.152.161.170] has quit [Ping timeout: 265 seconds]     
13:29 -!- pglazman [~pglazman@205.209.24.227] has joined #bitcoin-core-pr-reviews     
13:34 -!- pglazman [~pglazman@205.209.24.227] has quit [Quit: Textual IRC Client: www.textualapp.com]     
13:52 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
13:53 -!- orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Ping timeout: 268 seconds]     
14:00 -!- nadra [uid415365@gateway/web/irccloud.com/x-fjxllcozpkddcykh] has quit [Quit: Connection closed for inactivity]     
14:01 -!- csknk [~csknk@unaffiliated/csknk] has quit [Quit: leaving]     
14:03 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has quit [Read error: Connection reset by peer]     
14:14 -!- meshcollider [meshcollid@209.141.50.204] has quit [Remote host closed the connection]     
14:40 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Ping timeout: 265 seconds]     
14:42 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
14:46 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Remote host closed the connection]     
14:49 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
14:54 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Ping timeout: 272 seconds]     
15:05 -!- hanhua [68850952@104.133.9.82] has quit [Remote host closed the connection]     
16:03 -!- meshcollider [meshcollid@gateway/shell/ircnow/x-ljjxmkqlqsykocbq] has joined #bitcoin-core-pr-reviews     
16:32 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has quit [Ping timeout: 265 seconds]     
16:45 -!- meshcollider [meshcollid@gateway/shell/ircnow/x-ljjxmkqlqsykocbq] has quit [Quit: ZNC 1.7.4 - https://znc.in]     
16:53 -!- belcher [~belcher@unaffiliated/belcher] has quit [Quit: Leaving]     
16:58 -!- mol [~molly@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
17:05 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has joined #bitcoin-core-pr-reviews     
17:13 -!- dr-orlovsky [~dr-orlovs@77-58-192-184.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews     
17:14 -!- hebasto [~hebasto@95.164.65.194] has quit [Ping timeout: 268 seconds]     
17:33 -!- meshcollider [meshcollid@gateway/shell/ircnow/x-sbtjxckdujyurjue] has joined #bitcoin-core-pr-reviews     
18:49 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
18:53 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Ping timeout: 260 seconds]     
19:27 -!- hebasto [~hebasto@95.164.65.194] has joined #bitcoin-core-pr-reviews     
19:44 -!- achow101 [~achow101@unaffiliated/achow101] has quit [Ping timeout: 265 seconds]     
19:47 -!- felixfoertsch [~felixfoer@92.117.46.2] has joined #bitcoin-core-pr-reviews     
19:48 -!- achow101 [~achow101@unaffiliated/achow101] has joined #bitcoin-core-pr-reviews     
19:48 -!- felixfoertsch23 [~felixfoer@92.117.56.158] has quit [Ping timeout: 268 seconds]     
20:15 -!- molly [~molly@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
20:19 -!- mol [~molly@unaffiliated/molly] has quit [Ping timeout: 265 seconds]     
23:14 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
23:14 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has quit [Read error: Connection reset by peer]     
23:14 -!- jungly [~jungly@host4-0-dynamic.45-213-r.retail.telecomitalia.it] has joined #bitcoin-core-pr-reviews     
23:40 -!- vasild_ [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews     
23:43 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]     
23:51 -!- vasild_ is now known as vasild