--- Log opened Tue Jun 22 00:00:45 2021
00:01 -!- donny [uid133844@id-133844.brockwell.irccloud.com] has left #bitcoin-core-dev []
00:02 -!- sagi [~sagi@176.230.171.147] has quit [Ping timeout: 265 seconds]
00:03 -!- sagi [~sagi@176.230.171.147] has joined #bitcoin-core-dev
00:07 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
00:11 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 252 seconds]
00:35 -!- belcher_ is now known as belcher
00:43 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
00:46 -!- lkqwejhhgasdjhgn [~kljkljklk@p200300d46f03bc00c3c989967c4d9cc5.dip0.t-ipconnect.de] has joined #bitcoin-core-dev
00:47 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 268 seconds]
00:58 -!- evias_ [~evias__@196.red-88-6-131.staticip.rima-tde.net] has joined #bitcoin-core-dev
01:07 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
01:07 < bitcoin-git> [bitcoin] rebroad opened pull request #22307: Detect when GETBLOCKTXN requests are ignored. (master...DetectIngoredGetblocktxns) https://github.com/bitcoin/bitcoin/pull/22307
01:07 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
01:07 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
01:07 < bitcoin-git> [bitcoin] MarcoFalke opened pull request #22308: wallet: Add missing BlockUntilSyncedToCurrentChain (master...2106-wallet) https://github.com/bitcoin/bitcoin/pull/22308
01:07 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
01:09 -!- Guyver2 [Guyver@guyver2.xs4all.nl] has quit [Remote host closed the connection]
01:27 -!- gleb [~gleb@178.150.137.228] has quit [Ping timeout: 268 seconds]
01:34 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
01:39 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 265 seconds]
01:50 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
01:55 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 258 seconds]
02:02 -!- kexkey_ [~kexkey@static-198-54-132-110.cust.tzulo.com] has joined #bitcoin-core-dev
02:04 -!- kexkey [~kexkey@static-198-54-132-142.cust.tzulo.com] has quit [Ping timeout: 250 seconds]
02:08 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
02:08 < bitcoin-git> [bitcoin] MarcoFalke opened pull request #22309: blockstorage: Add missing atomic include (master...2106-include) https://github.com/bitcoin/bitcoin/pull/22309
02:08 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
02:17 < fanquake> wumpus / sipa:  can you please block Davidbrizio
02:22 < laanwj> fanquake: can't find the user
02:23 < fanquake> I think they've been deleted  already: https://github.com/Davidbrizio
02:24 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
02:28 -!- ChristianDecker[ is now known as cdecker[m]
02:28 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 252 seconds]
02:35 -!- evias_ [~evias__@196.red-88-6-131.staticip.rima-tde.net] has quit [Quit: This computer has gone to sleep]
02:43 -!- Kiminuo [~Kiminuo@141.98.103.76] has quit [Ping timeout: 246 seconds]
02:45 -!- pigeons [~pigeons@androzani.sysevolve.com] has quit [Ping timeout: 252 seconds]
02:45 -!- Kiminuo [~Kiminuo@141.98.103.76] has joined #bitcoin-core-dev
02:56 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
03:01 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 268 seconds]
03:05 -!- evias_ [~evias__@196.red-88-6-131.staticip.rima-tde.net] has joined #bitcoin-core-dev
03:06 -!- evias_ [~evias__@196.red-88-6-131.staticip.rima-tde.net] has quit [Client Quit]
03:07 -!- evias_ [~evias__@196.red-88-6-131.staticip.rima-tde.net] has joined #bitcoin-core-dev
03:17 -!- GIANTWORLDKEEPER [~pjetcetal@2.95.227.207] has quit [Quit: EXIT]
03:17 -!- GIANTWORLDKEEPER [~pjetcetal@2.95.227.207] has joined #bitcoin-core-dev
03:30 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
03:35 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 252 seconds]
03:50 -!- Evel-Knievel [~Evel-Knie@d5152f744.static.telenet.be] has quit [Ping timeout: 265 seconds]
03:53 -!- sagi [~sagi@176.230.171.147] has quit [Ping timeout: 252 seconds]
04:05 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
04:10 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 265 seconds]
04:13 -!- evias_ [~evias__@196.red-88-6-131.staticip.rima-tde.net] has quit [Quit: Leaving]
04:13 -!- evias [~evias__@196.red-88-6-131.staticip.rima-tde.net] has joined #bitcoin-core-dev
04:13 -!- evias [~evias__@196.red-88-6-131.staticip.rima-tde.net] has quit [Changing host]
04:13 -!- evias [~evias__@user/evias] has joined #bitcoin-core-dev
04:14 -!- Evel-Knievel [~Evel-Knie@d5152f744.static.telenet.be] has joined #bitcoin-core-dev
04:25 -!- Guest4 [~Guest4@45.67.96.239] has joined #bitcoin-core-dev
04:25 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
04:40 -!- Guest4 [~Guest4@45.67.96.239] has quit [Quit: Client closed]
04:41 -!- Guest4 [~Guest4@45.67.96.239] has joined #bitcoin-core-dev
04:41 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has joined #bitcoin-core-dev
04:43 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Ping timeout: 244 seconds]
04:43 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 258 seconds]
04:56 -!- yanmaani [~yanmaani@gateway/tor-sasl/yanmaani] has joined #bitcoin-core-dev
05:11 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
05:16 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 268 seconds]
05:34 -!- Guest4 [~Guest4@45.67.96.239] has quit [Quit: Client closed]
05:40 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has quit [Ping timeout: 252 seconds]
05:44 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
05:48 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 258 seconds]
06:16 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
06:21 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 265 seconds]
06:51 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
06:54 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
06:54 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/672870ab7ba5...327e2691f6e0
06:54 < bitcoin-git> bitcoin/master fa2d21f MarcoFalke: add missing atomic include
06:54 < bitcoin-git> bitcoin/master 327e269 MarcoFalke: Merge bitcoin/bitcoin#22309: blockstorage: Add missing atomic include
06:54 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
06:54 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
06:54 < bitcoin-git> [bitcoin] MarcoFalke merged pull request #22309: blockstorage: Add missing atomic include (master...2106-include) https://github.com/bitcoin/bitcoin/pull/22309
06:54 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
06:55 -!- Kiminuo [~Kiminuo@141.98.103.76] has quit [Ping timeout: 246 seconds]
06:55 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 252 seconds]
07:26 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
07:37 -!- kexkey [~kexkey@static-198-54-132-132.cust.tzulo.com] has joined #bitcoin-core-dev
07:38 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
07:38 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
07:40 -!- kexkey_ [~kexkey@static-198-54-132-110.cust.tzulo.com] has quit [Ping timeout: 258 seconds]
07:51 -!- gribble [~gribble@bitcoin/bot/gribble] has quit [Remote host closed the connection]
07:56 -!- gribble [~gribble@bitcoin/bot/gribble] has joined #bitcoin-core-dev
07:56 -!- mode/#bitcoin-core-dev [+o gribble] by ChanServ
08:00 -!- GIANTWORLDKEEPER [~pjetcetal@2.95.227.207] has quit [Quit: EXIT]
08:17 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has joined #bitcoin-core-dev
08:22 -!- GIANTWORLDKEEPER [~pjetcetal@2.95.227.207] has joined #bitcoin-core-dev
08:23 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has quit [Ping timeout: 268 seconds]
08:36 -!- Talkless [~Talkless@mail.dargis.net] has joined #bitcoin-core-dev
09:29 -!- lkqwejhhgasdjhgn [~kljkljklk@p200300d46f03bc00c3c989967c4d9cc5.dip0.t-ipconnect.de] has quit [Quit: Konversation terminated!]
09:44 -!- murch [~murch@user/murch] has joined #bitcoin-core-dev
09:46 -!- murch is now known as murchandamus
09:51 -!- sipsorcery [~sipsorcer@2a02:8084:6981:7880::3] has quit [Ping timeout: 264 seconds]
10:01 -!- evias [~evias__@user/evias] has quit [Quit: This computer has gone to sleep]
10:11 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has joined #bitcoin-core-dev
10:12 -!- jeremyr__ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
10:12 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Read error: Connection reset by peer]
10:14 -!- sipsorcery [~sipsorcer@2a02:8084:6981:7880::3] has joined #bitcoin-core-dev
10:18 -!- l3kk0 [~l3kk0@c-73-22-213-40.hsd1.il.comcast.net] has quit [Read error: Connection reset by peer]
10:18 -!- l3kk0 [~l3kk0@TASTYWORKS.ear2.Chicago2.Level3.net] has joined #bitcoin-core-dev
10:21 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
10:21 < bitcoin-git> [bitcoin] ariard opened pull request #22310: test: Add functional test for replacement penalty check  (master...2021-06-add-rbf5-test) https://github.com/bitcoin/bitcoin/pull/22310
10:21 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
10:26 -!- reez [~textual@45-30-22-183.lightspeed.nsvltn.sbcglobal.net] has joined #bitcoin-core-dev
10:26 -!- tripleslash is now known as imsaguy
10:26 -!- imsaguy is now known as tripleslash
10:26 -!- tripleslash is now known as [\\\]
10:26 -!- reez [~textual@45-30-22-183.lightspeed.nsvltn.sbcglobal.net] has quit [Client Quit]
10:26 -!- reez [~textual@45-30-22-183.lightspeed.nsvltn.sbcglobal.net] has joined #bitcoin-core-dev
10:31 -!- reez [~textual@45-30-22-183.lightspeed.nsvltn.sbcglobal.net] has quit [Ping timeout: 258 seconds]
10:45 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
10:45 < bitcoin-git> [bitcoin] MarcoFalke opened pull request #22311: test: Add missing syncwithvalidationinterfacequeue in p2p_blockfilters (master...2106-testsyncwithvalidationinterfacequeue) https://github.com/bitcoin/bitcoin/pull/22311
10:45 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
10:49 -!- emcy_ [~emcy@user/emcy] has quit [Quit: Leaving]
10:49 -!- emcy [~emcy@user/emcy] has joined #bitcoin-core-dev
11:00 -!- emcy [~emcy@user/emcy] has quit [Quit: Leaving]
11:02 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has quit [Ping timeout: 258 seconds]
11:04 -!- Guyver2 [Guyver@guyver2.xs4all.nl] has joined #bitcoin-core-dev
11:09 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
11:09 < bitcoin-git> [bitcoin] siv2r opened pull request #22312: changes for wait_for_getheaders to include hash_list (master...modify-wait-getheaders) https://github.com/bitcoin/bitcoin/pull/22312
11:09 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
11:15 -!- sipsorcery [~sipsorcer@2a02:8084:6981:7880::3] has quit [Ping timeout: 268 seconds]
11:15 -!- belcher [~belcher@user/belcher] has quit [Quit: Leaving]
11:17 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
11:17 < bitcoin-git> [bitcoin] MarcoFalke opened pull request #22313: test: Add missing sync_all to feature_coinstatsindex (master...2106-testSync) https://github.com/bitcoin/bitcoin/pull/22313
11:17 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
11:22 -!- bomb-on [~bomb-on@194.144.47.113] has joined #bitcoin-core-dev
11:29 -!- emcy [~emcy@user/emcy] has joined #bitcoin-core-dev
11:29 -!- Talkless [~Talkless@mail.dargis.net] has quit [Quit: Konversation terminated!]
11:41 -!- sipsorcery [~sipsorcer@2a02:8084:6981:7880::3] has joined #bitcoin-core-dev
11:45 -!- jeremyr__ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
11:51 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
11:52 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
11:52 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
11:56 < jamesob> I know this is an age old and time honored tradition, but just for fun: when's the last time someone looked seriously at replacing leveldb with something bespoke for our access patterns?
11:56 < jamesob> not saying it's something I want to do but may be a fun project for a new contributor/summer intern or something who's so inclined
11:57 < jamesob> even having a relatively thorough enumeration of those access patterns and what exactly we need from such a library would be interesting
11:58 < laanwj> having the entire utxo set in memory still works best :-)
11:59 -!- evias [~evias__@user/evias] has joined #bitcoin-core-dev
12:02 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has joined #bitcoin-core-dev
12:03 < laanwj> could just write it out as a linear file on shutdown (and read it in on startup); the drawback of not using a database is, besides the memory use, that unexpected crashes of the daemon lose the entire state, as it cannot be written incrementally
12:03 < jamesob> laanwj: absolutely :)
12:03 < jamesob> but ofc we don't want to make running a full node require 10gb of memory or whatever it is these days
12:04 < laanwj> probably not, and after the initial sync the performance tradeoff becomes different anyway
12:06 < laanwj> after that there's pretty much two use cases, either you want verification to go as quickly as possible (e.g. miners), which warrants keeping everything in memory, or the speed is pretty much irrelevalt (personal nodes)
12:07 < laanwj> in which case leveldb is good enough?
12:09 < jeremyru_> jamesob: i think it'd be fun to have some decidedly *worse* databases too -- e.g. a Filesystem tree 
12:09 -!- Guest96 [~Guest96@2001:861:38c7:a830:19e0:63a5:7372:46aa] has joined #bitcoin-core-dev
12:10 < jamesob> jeremyru_: totally; would be fun to see to what extent that degrades performance
12:11 < jamesob> laanwj: right absolutely. I guess I'm just compelled to think about it because I wonder if we couldn't come up with something equally robust/performant but simpler. After dealing with the issue underlying #22263 I was reminded that leveldb definitely isn't perfect and drags in some stuff we may not need (e.g. snaphots)
12:11 < jeremyru_> it'd also be interesting to put in sqlite because then you can build out more indexing stuff, and sqlite is already a dependency for wallet
12:11 <@gribble> https://github.com/bitcoin/bitcoin/issues/22263 | refactor: wrap CCoinsViewCursor in unique_ptr by jamesob · Pull Request #22263 · bitcoin/bitcoin · GitHub
12:12 < laanwj> back in the day we tried some experiments with LMDB but while read performance was somewhat better, write performance was worse
12:12 < jamesob> jeremyru_: problem with that is that would drag sqlite into consensus which we definitely don't want
12:12 < laanwj> snapshots are useful for being able to run utxo statistics or backup in the background
12:13 < jamesob> yeah that's a good point, and probably not something we'd want to implement ourselves
12:13 < jeremyru_> jamesob: true, just thinking more generally about things a node operator might want to have, as experimental/run as an internal API node stuff
12:13 < laanwj> i doubt we need more indexing stuff, a key/value database is fine for utxos
12:14 < jamesob> right
12:14 < sipa> we tried sqlite
12:14 < sipa> at the time it had terrible performance for this kind of load
12:14 < laanwj> yes
12:14 < sipa> i don't think that has changed; it's just not designed for this
12:15 < jeremyru_> having everything in memory performance wise should be doable mostly as command line options right on cache sizes? I guess the flushes need to be sync on some responses.
12:16 < sipa> if you run with -dbcache=infinity and reindex you'll effectively do the entire sync without a single db write
12:16 < jamesob> jeremyru_: mem only a nonstarter for reasons laanwj mentioned; need to have durability for crashes
12:16 < sipa> (the blocks will be written to disk, but no flushes would occur)
12:16 < jeremyru_> sipa: what happens on crash?
12:16 < laanwj> jamesob: I mean it's fine for specific scenarios where you have UPS backup, or fallback nodes
12:16 < jamesob> right but we can't assume that as a generality ofc
12:17 < laanwj> sure, but if you want to specialize for use cases
12:17 < sipa> jeremyru_: you'll start over from scratch
12:17 < jamesob> nor can we assume a lot of RAM... but I do like this idea of an optional "sync as fast as you can" mode that ensures sufficient memory and just blazes through an ibd
12:17 < sipa> jamesob: that exists, just set dbcache to all your memory
12:17 < jamesob> (not to mention makes tip maintenance as fast as possible for miners)
12:18 < jamesob> sipa: right
12:18 < jamesob> sipa: although I guess such a mode would preclude periodic flushing?
12:18 < jeremyru_> jamesob that should give you a good estimate on how much gain to be had
12:18 < laanwj> sipa: yes, it just lacks a way to read in the entire database at node restart at the moment
12:19 < sipa> laanwj: right, that would be easy to add
12:19 < laanwj> sure
12:19 < jeremyru_> jamesob: you could make a double buffered thing if memory really = infinity 
12:20 < jamesob> jeremyru_: right
12:20 < jeremyru_> so that you just copy the entire utxo cache, and then write the snapshot periodically
12:20 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
12:20 < bitcoin-git> [bitcoin] nourou4them opened pull request #22314: doc: Install WSL on non-system drive and compile Bitcoin (master...patch-1) https://github.com/bitcoin/bitcoin/pull/22314
12:20 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
12:20 < jeremyru_> avoid the restart from scratch issue
12:20 < jeremyru_> or you could add another cache layer temoporarily
12:21 < jeremyru_> so that all the reads/writes while flushing are temporarily against another layer while you flush
12:22 < sipa> that's surprisingly hard to do right
12:22 < sipa> in combination with the "delete entries that are spent if they have been created before the last flush"
12:22 < sipa> optimization (which we benefit massively from, as all utxos are written once, read once, deleted once)
12:24 < jamesob> haha you know I'm now remembering why I arrived at "screw all this ~10% optimization stuff, let's just work on assumeutxo"
12:25 -!- murchandamus is now known as murch
12:26 < jeremyru_> jamesob: hmm
12:26 < jeremyru_> one idea:
12:26 < jeremyru_> just write your assumeutxo rolling hashes to disk as you go
12:26 < jeremyru_> fully verified, but you could re download if you corrupt?
12:27 < jamesob> that's kind of interesting, but you've still gotta keep the block data (and index) somewhere
12:27 < jamesob> anyway sounds precarious/complex relative to the benefits
12:27 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
12:27 < bitcoin-git> [gui] luke-jr opened pull request #366: Dark Mode fixes/portability (master...gui_palettechange) https://github.com/bitcoin-core/gui/pull/366
12:27 < jamesob> I think it's one thing if there's a sizable optimization we can make for tip maintenance (like avoiding synchronous flushes), but otherwise maybe not worth it to get too fancy
12:27 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
12:29 < sipa> i had a design a few years ago that would permit concurrent flushing with cache updating (so it doesn't need stop-the-clock and wipe-all-memory on every flush), but it was pretty nontrivial
12:30 < sipa> while still maintaining the soon-spend-never-hits-disk optimization within some window of blocks
12:32 < jamesob> also the long-neglected #17487 seems pretty relevant here
12:32 <@gribble> https://github.com/bitcoin/bitcoin/issues/17487 | coins: allow write to disk without cache drop by jamesob · Pull Request #17487 · bitcoin/bitcoin · GitHub
12:33 < jamesob> > the soon-spend-never-hits-disk optimization 
12:33 < jamesob> yeah this one is pretty interesting, feel like there's some potential there
12:34 < sipa> well, we already use it
12:34 < sipa> it's the reason why -dbcache=huge is so much faster than smaller caches
12:34 < sipa> it's not just avoiding reads from disk - it's mostly preventing things from being written in the first place
12:35 < sipa> but combining it with partial flushing is hard, because it leaves you in an inconsistent state
12:35 < sipa> every utxo individually on disk will be consistent with the state it had at some point between the last fully flushed block, and the last processed block
12:35 < sipa> but you can't guarantee they're all consistent with each other
12:38 < jamesob> so basically you'd have to use an ordered map for cacheCoins to do any better than we're doing right now I think, right? or have some index for insertion order
12:39 < sipa> insertion order does not help
12:39 < sipa> or at least, not on its own
12:39 < jamesob> will younger coins are more likely to be spent, right?
12:39 < jamesob> *well
12:39 < jeremyru_> if you were to, say, download N blocks at a time, you could make a lookahead cache that tells you what you should flush to disk and what will be deleted before you actually process the blocks.
12:39 < sipa> the problem is that the order in which you delete never-written-to-disk entries from your cache isn't the same as the order the utxos are created it
12:39 < jamesob> jeremyru_: there's some kind of optimization like that the utreexo guys are doing
12:39 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has quit [Ping timeout: 252 seconds]
12:40 < sipa> jamesob: oh you just mean as an access optimization? i don't think that's the bottleneck
12:40 < jamesob> sipa: no, I mean when partial flushing, avoid flushing coins that are likely soon to be spent
12:40 < sipa> jamesob: there is no solution for that problem
12:40 < sipa> you just need to keep track of the range of blocks that your state on disk corresponds to
12:41 < sipa> and on restart, reprocess those blocks to fix the db
12:41 < sipa> we do that already btw, to a limited extent, if you crash in the middle of a disk flush
12:41 < jamesob> right
12:41 < sipa> but doing it asynchronously makes tracking a lot harder, because you can have reorgs mixing different histories that all get written to disk at once
12:42 < sipa> (and triggering a full flush on reorg would be disasterous for orphan rates)
12:42 < sipa> right now we only have an inconsistent state on disk from the moment a flush begins until the point where it finishes
12:43 < sipa> continuously partial flushing (which would be far better for performance) would mean you're effectively *always* inconsistent on disk, but need to retain the ability to recover from it
12:44 < jeremyru_> sipa: if crash during that do we corrupt recoverably? detectably? or just requires reindex?
12:44 < sipa> jeremyru_: it gets detected at startup, and reliably recovered without full reindex
12:44 < sipa> (unless you have disk errors of course)
12:45 < sipa> at the start of a flush, a record is written of the form "sync started, block range A...B" where B is the current tip, and A is the old tip that was flushed to
12:45 < sipa> at the end, that record is removed and replaced with "synced to block B"
12:46 < sipa> at startup, if a range block is present, those blocks are processed again, and their UTXO updated are applied to disk, without any other validation
12:49 < sipa> one problem with this is that due to fewer consistency guarantees at that point, certain optimizations can't be used, and this reprocessing is surprisingly slower than actual validation
12:49 < sipa> so if the range is too big, it's actually slower than just a reindex
13:19 -!- belcher [~belcher@user/belcher] has joined #bitcoin-core-dev
13:33 < dhruv> BIP324 question: Does the 3-byte encrypted length in the proposed AEAD construct include the 3 bytes of the length itself? The test vector here seems to imply so but it would be nice to confirm: https://github.com/bitcoin/bitcoin/blob/327e2691f6e0498aa868af965bb4a951c5be17c5/src/test/crypto_tests.cpp#L710
13:35 < dongcarl> meshcollider: Do you remember why you added a -r flag here: https://github.com/bitcoin-core/gitian.sigs/blob/dfaecb5fa5062a3b2443e1f93933139b864745d8/scripts/files-touched-check.py#L16
13:35 < dongcarl> I don't see it documented anywhere
13:38 -!- l3kk0 [~l3kk0@TASTYWORKS.ear2.Chicago2.Level3.net] has quit [Read error: Connection reset by peer]
13:38 -!- l3kk0 [~l3kk0@TASTYWORKS.ear2.Chicago2.Level3.net] has joined #bitcoin-core-dev
13:43 -!- l3kk0 [~l3kk0@TASTYWORKS.ear2.Chicago2.Level3.net] has quit [Ping timeout: 265 seconds]
13:43 -!- l3kk0 [~l3kk0@c-73-22-213-40.hsd1.il.comcast.net] has joined #bitcoin-core-dev
13:46 -!- naiza [~naiza@180.188.251.115] has joined #bitcoin-core-dev
13:54 -!- evias [~evias__@user/evias] has quit [Quit: This computer has gone to sleep]
14:01 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
14:01 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
14:02 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
14:02 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
14:10 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
14:10 < bitcoin-git> [bitcoin] MarcoFalke pushed 3 commits to master: https://github.com/bitcoin/bitcoin/compare/327e2691f6e0...c93e123dc72b
14:10 < bitcoin-git> bitcoin/master fadb550 MarcoFalke: wallet: Add missing BlockUntilSyncedToCurrentChain
14:10 < bitcoin-git> bitcoin/master fa27baa MarcoFalke: Revert "test: Add temporary logging to debug #20975"
14:10 < bitcoin-git> bitcoin/master c93e123 MarcoFalke: Merge bitcoin/bitcoin#22308: wallet: Add missing BlockUntilSyncedToCurrent...
14:10 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
14:11 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
14:11 < bitcoin-git> [bitcoin] MarcoFalke merged pull request #22308: wallet: Add missing BlockUntilSyncedToCurrentChain (master...2106-wallet) https://github.com/bitcoin/bitcoin/pull/22308
14:11 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
14:16 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
14:17 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
14:20 -!- Guyver2 [Guyver@guyver2.xs4all.nl] has quit [Quit: Going offline, see ya! (www.adiirc.com)]
14:26 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
14:26 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
14:48 -!- jeremyr__ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
14:48 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Read error: Connection reset by peer]
14:52 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
14:52 -!- jeremyr__ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Read error: Connection reset by peer]
15:00 -!- Guest96 [~Guest96@2001:861:38c7:a830:19e0:63a5:7372:46aa] has quit [Quit: Client closed]
15:03 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
15:20 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
15:21 -!- smartin [~Icedove@88.135.18.171] has quit [Quit: smartin]
15:26 -!- naiza [~naiza@180.188.251.115] has quit [Quit: Client closed]
15:41 -!- GIANTWORLDKEEPER [~pjetcetal@2.95.227.207] has quit [Quit: EXIT]
15:44 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 252 seconds]
15:45 -!- GIANTWORLDKEEPER [~pjetcetal@2.95.227.207] has joined #bitcoin-core-dev
15:47 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has quit [Remote host closed the connection]
15:48 -!- sipsorcery [~sipsorcer@2a02:8084:6981:7880::3] has quit [Ping timeout: 268 seconds]
16:02 < darosior> morcos: why does the fee estimator disregard CPFP? You mentioned in https://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg06405.html that it would skew estimates, but how so? I don't have statistics but i think CPFP is widely used on the network and might be even more in the future with package relay and it becoming a first
16:02 < darosior> class citizen in L2 protocols. It seems to me it could bias estimates downward as you would see a low-fee transaction (or actually lot of them) being confirmed quickly whereas you should not rely on their feerate as a decent estimate for being confirmed quickly. Accounting for their parent might be conservative, and accounting for it as a package
16:02 < darosior> would be accurate
16:02 < darosior> s/for their parent/for their child/
16:13 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
16:18 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 268 seconds]
16:19 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
16:20 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has joined #bitcoin-core-dev
16:21 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
16:23 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
16:25 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has quit [Ping timeout: 268 seconds]
16:29 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
16:43 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
16:56 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has joined #bitcoin-core-dev
17:01 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has quit [Ping timeout: 268 seconds]
17:07 -!- belcher_ [~belcher@user/belcher] has joined #bitcoin-core-dev
17:11 -!- belcher [~belcher@user/belcher] has quit [Ping timeout: 252 seconds]
17:17 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
17:20 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
17:21 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has joined #bitcoin-core-dev
17:24 -!- S3RK [~S3RK@user/s3rk] has quit [Ping timeout: 272 seconds]
17:26 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has quit [Ping timeout: 252 seconds]
17:30 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
17:30 < bitcoin-git> [bitcoin] prayank23 opened pull request #22316: doc: Add 5 privacy recommendations in tor.md (master...tor-privacy-recommend) https://github.com/bitcoin/bitcoin/pull/22316
17:30 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
17:30 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
17:30 < bitcoin-git> [bitcoin] prayank23 opened pull request #22317: doc: Highlight DNS requests part in tor.md (master...highlight-dns-request) https://github.com/bitcoin/bitcoin/pull/22317
17:30 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
17:41 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
17:48 < luke-jr> how to restart bitcoinbuilds CI jobs? https://bitcoinbuilds.org/?job=8eba38f6-e542-4497-9330-d9a0f1183f80 seems stalled for nothing related to PR
17:50 -!- S3RK [~S3RK@user/s3rk] has joined #bitcoin-core-dev
17:50 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
17:56 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
18:19 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
18:19 < bitcoin-git> [bitcoin] prayank23 closed pull request #21157: doc: Improve Tor docs (master...tor-docs-add) https://github.com/bitcoin/bitcoin/pull/21157
18:19 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
18:20 -!- jesseposner [~jesse@2601:647:0:89:6de9:9aa6:8bb7:1691] has quit [Ping timeout: 250 seconds]
18:21 -!- jesseposner [~jesse@2601:647:0:89:dcb5:c682:ac61:4798] has joined #bitcoin-core-dev
19:11 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
19:11 < bitcoin-git> [bitcoin] hebasto opened pull request #22318: build: Avoid @GLIBC_2.25 symbols for compatibility (master...210623-random) https://github.com/bitcoin/bitcoin/pull/22318
19:11 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
19:13 -!- jarthur [~jarthur@2603-8080-1540-002d-f9cc-6895-8b3d-e0f4.res6.spectrum.com] has quit [Quit: jarthur]
19:13 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
19:14 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Remote host closed the connection]
19:16 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has joined #bitcoin-core-dev
19:20 -!- bomb-on [~bomb-on@194.144.47.113] has quit [Quit: aллилѹіа!]
19:21 -!- jeremyru_ [~jeremyrub@024-176-247-182.res.spectrum.com] has quit [Ping timeout: 265 seconds]
19:23 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has joined #bitcoin-core-dev
19:27 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has quit [Ping timeout: 265 seconds]
19:53 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has joined #bitcoin-core-dev
20:03 -!- stonefox [~stonefox@user/stonefox] has joined #bitcoin-core-dev
20:04 -!- pigeons [~pigeons@androzani.sysevolve.com] has joined #bitcoin-core-dev
20:21 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has quit [Ping timeout: 252 seconds]
20:24 < meshcollider> dongcarl: it's been a while but it may be a mistake, I think -r is a diff-tree flag not a diff flag
20:25 < meshcollider> To recurse into subtrees
20:42 -!- baldur [~baldur@pool-108-30-51-126.nycmny.fios.verizon.net] has quit [Ping timeout: 244 seconds]
20:55 -!- baldur [~baldur@pool-108-30-51-126.nycmny.fios.verizon.net] has joined #bitcoin-core-dev
21:23 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has joined #bitcoin-core-dev
21:28 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has quit [Ping timeout: 258 seconds]
22:33 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has joined #bitcoin-core-dev
23:08 -!- dermoth_ [~dermoth@user/dermoth] has joined #bitcoin-core-dev
23:08 -!- dermoth [~dermoth@user/dermoth] has quit [Killed (NickServ (GHOST command used by dermoth_))]
23:08 -!- dermoth_ is now known as dermoth
23:09 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
23:09 < bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/c93e123dc72b...03aa59a4e58f
23:09 < bitcoin-git> bitcoin/master fafd916 MarcoFalke: test: Add missing sync_all to feature_coinstatsindex
23:09 < bitcoin-git> bitcoin/master 03aa59a MarcoFalke: Merge bitcoin/bitcoin#22313: test: Add missing sync_all to feature_coinsta...
23:09 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
23:10 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
23:10 < bitcoin-git> [bitcoin] MarcoFalke merged pull request #22313: test: Add missing sync_all to feature_coinstatsindex (master...2106-testSync) https://github.com/bitcoin/bitcoin/pull/22313
23:10 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
23:21 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has quit [Ping timeout: 258 seconds]
23:24 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has joined #bitcoin-core-dev
23:29 -!- AaronvanW [~AaronvanW@71pc74.sshunet.nl] has quit [Ping timeout: 265 seconds]
23:34 -!- smartin [~Icedove@88.135.18.171] has joined #bitcoin-core-dev
23:54 -!- sagi [~sagi@bzq-79-179-156-161.red.bezeqint.net] has joined #bitcoin-core-dev
23:56 -!- cfields [~cfields@207.172.223.102] has quit [Read error: Connection reset by peer]
23:57 -!- cfields [~cfields@207.172.223.102] has joined #bitcoin-core-dev
--- Log closed Wed Jun 23 00:00:46 2021