--- Day changed Thu May 07 2020
00:12 -!- bordalix [~bordalix@191.87.108.93.rev.vodafone.pt] has joined #bitcoin-core-pr-reviews     
00:17 -!- bordalix [~bordalix@191.87.108.93.rev.vodafone.pt] has quit [Ping timeout: 256 seconds]     
00:23 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
00:27 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has quit [Ping timeout: 264 seconds]     
00:30 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:cc78:c0db:3f4d:fe88] has joined #bitcoin-core-pr-reviews     
00:32 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:cc78:c0db:3f4d:fe88] has quit [Client Quit]     
00:40 -!- bordalix [~bordalix@191.87.108.93.rev.vodafone.pt] has joined #bitcoin-core-pr-reviews     
00:43 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:6d15:199d:f584:8f52] has joined #bitcoin-core-pr-reviews     
00:45 -!- bordalix [~bordalix@191.87.108.93.rev.vodafone.pt] has quit [Ping timeout: 260 seconds]     
00:46 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:6d15:199d:f584:8f52] has quit [Client Quit]     
00:54 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
00:56 -!- Netsplit *.net <-> *.split quits: jkczyz, peltre
00:57 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 272 seconds]     
01:01 -!- peltre [sid268329@gateway/web/irccloud.com/x-gqpwhefsxogbyghd] has joined #bitcoin-core-pr-reviews     
01:01 -!- jkczyz [sid419941@gateway/web/irccloud.com/x-lrtiexgadoqaaabo] has joined #bitcoin-core-pr-reviews     
01:02 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:6d15:199d:f584:8f52] has joined #bitcoin-core-pr-reviews     
01:02 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:6d15:199d:f584:8f52] has quit [Client Quit]     
01:21 -!- bordalix [~bordalix@191.87.108.93.rev.vodafone.pt] has joined #bitcoin-core-pr-reviews     
01:23 -!- peltre [sid268329@gateway/web/irccloud.com/x-gqpwhefsxogbyghd] has quit [Max SendQ exceeded]     
01:23 -!- peltre [sid268329@gateway/web/irccloud.com/x-efiezggjkjikakuw] has joined #bitcoin-core-pr-reviews     
01:26 -!- bordalix [~bordalix@191.87.108.93.rev.vodafone.pt] has quit [Ping timeout: 256 seconds]     
01:38 -!- bordalix [~bordalix@191.87.108.93.rev.vodafone.pt] has joined #bitcoin-core-pr-reviews     
01:53 -!- bordalix [~bordalix@191.87.108.93.rev.vodafone.pt] has quit []     
02:24 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
02:28 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has quit [Ping timeout: 246 seconds]     
02:39 -!- icota[m] [icotamatri@gateway/shell/matrix.org/x-inrfpbneyedsmhht] has quit [Quit: killed]     
02:43 < jonatack> ja: afaik this channel isn't logged, unless that changed
02:55 -!- icota[m] [icotamatri@gateway/shell/matrix.org/x-ferchfbwyendkkci] has joined #bitcoin-core-pr-reviews     
03:44 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection]     
03:45 -!- grunch__ [~grunch@2800:810:547:c46:25e8:a7de:f130:fe8] has joined #bitcoin-core-pr-reviews     
03:45 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-core-pr-reviews     
03:53 < jonatack> I updated the article on how to compile bitcoin core https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests
03:54 < jonatack> diff here: https://gist.github.com/jonatack/a1d4f3582c2c6577cf1b7abc79a0cb92
04:03 -!- molz_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
04:06 -!- mol_ [~mol@unaffiliated/molly] has quit [Ping timeout: 260 seconds]     
04:24 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
04:29 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has quit [Ping timeout: 264 seconds]     
04:31 < brikk> jonatack: thanks!
04:50 -!- jonatack_ [~jon@37.164.233.66] has joined #bitcoin-core-pr-reviews     
04:53 < kallewoof> belated congrats on 1 year, everyone. :)
04:54 -!- jonatack [~jon@37.173.38.62] has quit [Ping timeout: 264 seconds]     
05:06 -!- theStack [~honeybadg@vps1648322.vs.webtropia-customer.com] has joined #bitcoin-core-pr-reviews     
05:23 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
05:26 -!- molz_ [~mol@unaffiliated/molly] has quit [Ping timeout: 240 seconds]     
06:25 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
06:38 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has quit [Ping timeout: 256 seconds]     
06:45 -!- mol_ [~mol@unaffiliated/molly] has quit [Ping timeout: 264 seconds]     
07:00 -!- brakmic [~brakmic@185.183.85.108] has joined #bitcoin-core-pr-reviews     
07:11 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
07:16 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has quit [Ping timeout: 246 seconds]     
07:20 -!- thomasb06 [~user@eth-west-pareq2-46-193-0-224.wb.wifirst.net] has joined #bitcoin-core-pr-reviews     
07:21 < thomasb06> jonatack_: hey. I may have a suggestion of improvement for your guide to compile
07:21 < thomasb06> it seems that "export BDB_PREFIX='<PATH-TO>/db4'"
07:22 < thomasb06> takes the absolute path to db4
07:26 < jonatack_> thomasb06: sure, it's whatever path you want to use, absolute or relative, up to you. The installation script in /contrib provides an absolute path iirc, but you can use a relative one if you prefer.
07:27 < thomasb06> well, it didn't work with the relative path:
07:27 < thomasb06> I used './db4' and
07:28 < thomasb06> it gave:
07:28 < thomasb06>   CXXLD    bitcoind
07:28 < thomasb06> ../libtool: line 7703: cd: ./db4/lib: No such file or directory
07:28 < thomasb06> libtool:   error: cannot determine absolute directory name of './db4/lib'
07:28 < thomasb06> make[2]: *** [Makefile:7387: bitcoind] Error 1
07:28 < thomasb06> make[2]: Leaving directory '/home/user/github/bitcoin/src'
07:28 < thomasb06> make[1]: *** [Makefile:17329: all-recursive] Error 1
07:28 < thomasb06> make[1]: Leaving directory '/home/user/github/bitcoin/src'
07:28 < thomasb06> make: *** [Makefile:780: all-recursive] Error 1
07:28 < thomasb06>       
07:30 < jonatack_> a relative path will depend from where you call it for the build, so i would likely avoid doing that
07:31 < jonatack_> i set it as a bash alias in accordance with the absolute path to /db4, and done
07:31 < thomasb06> it was from the bitcoin/ directory
07:31 < thomasb06> but with the absolute path it goes well
07:45 -!- mol [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
07:51 < thomasb06> jonatack_: the build takes ages... What option could I have used to get the most out of the processor?
07:52 < thomasb06> only one core is busy at 100%
07:53 < jonatack_> the first time is slow. ccache speeds up subsequent builds. see the details and links in that web page ;)
07:53 < sipa> the -j option make sets the number of threads
07:54 < jonatack_> make sure you use -j as well, it's described in the web page
07:55 < thomasb06> then, the right command would have been: make -j"$(($(nproc)+1))"
07:55 < thomasb06> I'll know for the next time
07:56 < jonatack_> if you're on linux, yes
07:56 < thomasb06> under Arch...
07:57 < thomasb06> (wow... That's what I'm talking about)
07:57 < sipa> or you could remember how many cpus you have and add 1
07:57 < jonatack_> sipa: (added the bdb flag option to the doc, if you have further feedback i'm happy to update it)
07:57 < sipa> make -j9 if you have an 8-thread system
07:58 < thomasb06> sipa: that's 4, so make -j4
07:58 < sipa> thomasb06: add 1
07:58 < thomasb06> oops, make -j5...
07:59 < thomasb06> (arf, with the -j option the compilation took only seconds)
08:00 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
08:04 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has quit [Ping timeout: 246 seconds]     
08:05 < hebasto> sipa: could make -j"$(($(nproc)+1))" cause oversubscription in OS scheduler?
08:07 < jonatack_> hebasto: i've heard some devs don't add 1 to nproc because it slows other operations too much during the builds
08:08 < thomasb06> a test has failed:
08:08 < thomasb06> Running tests: hash_tests from test/hash_tests.cpp
08:08 < thomasb06> ../build-aux/test-driver: line 107: 551142 Aborted                 (core dumped) "$@" > $log_file 2>&1
08:08 < thomasb06> FAIL: qt/test/test_bitcoin-qt
08:08 < thomasb06>       
08:09 < sipa> hebasto: the theory is that not every build thread is continuously using CPU; there are bursts of more IO-heavy operations as well; adding one allows an extra thread to use the times the other N are not busy
08:12 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:6c19:bdb9:a429:1127] has joined #bitcoin-core-pr-reviews     
08:14 < hebasto> sipa: thanks!
08:16 < thomasb06> regarding the .py tests, many have a 'Error: no RPC connection' message. It takes to be connected to other nodes?
08:23 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
08:25 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:6c19:bdb9:a429:1127] has quit [Quit: Sleep mode]     
08:26 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:6c19:bdb9:a429:1127] has joined #bitcoin-core-pr-reviews     
08:30 < jonatack_> thomasb06: rebuilding after running make clean or make distclean might help (mentioned in the article for when you see issues)
08:32 < thomasb06> jonatack_: since it is my first build, there should be nothing to clean?
08:42 -!- grunch__ [~grunch@2800:810:547:c46:25e8:a7de:f130:fe8] has quit [Ping timeout: 246 seconds]     
08:49 -!- Talkless [~Talkless@hst-227-49.splius.lt] has joined #bitcoin-core-pr-reviews     
08:59 -!- andrewtoth_ [~andrewtot@gateway/tor-sasl/andrewtoth] has joined #bitcoin-core-pr-reviews     
08:59 -!- andrewtoth [~andrewtot@gateway/tor-sasl/andrewtoth] has quit [Ping timeout: 240 seconds]     
09:00 -!- andrewtoth_ [~andrewtot@gateway/tor-sasl/andrewtoth] has quit [Remote host closed the connection]     
09:00 < jnewbery> We'll get started on https://github.com/bitcoin/bitcoin/pull/18877 in an hour
09:02 -!- michaelf_ [~textual@2a00:23c5:be01:b201:d011:ff30:38d9:30c9] has joined #bitcoin-core-pr-reviews     
09:04 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:6c19:bdb9:a429:1127] has quit [Ping timeout: 240 seconds]     
09:13 -!- grunch__ [~grunch@2800:810:547:c46:25e8:a7de:f130:fe8] has joined #bitcoin-core-pr-reviews     
09:19 -!- vasild_ [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews     
09:23 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]     
09:23 -!- vasild_ is now known as vasild     
09:40 < jkczyz> I have a conflict today but may be able to join late
09:45 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection]     
09:50 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-core-pr-reviews     
09:50 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 260 seconds]     
09:54 < ja> jonatack_: /topic says the channel is logged. i am quoting topic. if topic is wrong, topic should be changed.
09:59 -!- lightlike [~lightlike@p200300C7EF1D8900B0A411C1CCBBCDEB.dip0.t-ipconnect.de] has joined #bitcoin-core-pr-reviews     
10:00 < jnewbery> #startmeeting
10:00 < jnewbery> hi folks, welcome to Review Club special BIP 157 edition :)
10:00 < theStack> hi!     
10:00 < fjahr> hi     
10:00 < gleb> hi     
10:01 < nehan> hi     
10:01 < jonatack_> ja: i think that was added to /topic because the very first nothing was not logged, and then when it was decided to add the meeting logs to the website, the warning was added.
10:01 -!- jonatack_ [~jon@37.164.233.66] has quit [Quit: jonatack_]     
10:01 -!- jonatack [~jon@37.164.233.66] has joined #bitcoin-core-pr-reviews     
10:01 < jnewbery> Like I metioned yesterday, I'm making an effort to get the BIP 157 work done by jimpo merged. Tracking PR is here: https://github.com/bitcoin/bitcoin/pull/18876
10:01 -!- michaelf_ [~textual@2a00:23c5:be01:b201:d011:ff30:38d9:30c9] has quit [Ping timeout: 240 seconds]     
10:01 < jonatack> hi     
10:02 < jnewbery> I've split it up into more easily reviewable chunks. The first is here: https://github.com/bitcoin/bitcoin/pull/18877
10:02 < jnewbery> This is slightly different from other review club meetings. I haven't written notes and questions, but I wanted to make a space available for everyone to ask questions and really get deep into the implementation.
10:03 < jnewbery> We did review the full PR about 6 months ago. pinheadmz did a great job hosting: https://bitcoincore.reviews/16442.html
10:03 < jnewbery> so if you need a refresher on what the full PR is doing, I recommend re-reading those notes and logs
10:04 < jnewbery> The first PR I split out is just serving cfcheckpts, and doesn't signal NODE_COMPACT_FILTERS in the version message
10:04 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:358b:f356:e6b1:9e1b] has joined #bitcoin-core-pr-reviews     
10:04 < jnewbery> it's a very limited subset of the functionality, but it can be used for testing and serving checkpts to pre-configured clients
10:05 < jnewbery> Does anyone have any questions about the full PR or the sub-PR?
10:05 < theStack> jnewbery: any concrete reason why you chose the handling of (get)cfcheckpts as first reviewable chunk? is it the simplest one?
10:06 < nehan> jnewbery: i am also interested in how you decided to break of this piece
10:06 < nehan> *off
10:06 < jnewbery> theStack: to some extent yes. I expect the normal client operation will be: 1. fetch cfcheckpts, 2. fetch all other cfheaders, 3. fetch cfilters.
10:07 < jnewbery> That's how the test is structured at least
10:07 < jnewbery> so in that order of operations, supporting getcfcheckpt and sending the cfcheckpt message is the first thing to do
10:07 < fjahr> It's kind of nit but I am a bit annoyed that it's sometimes block filter and sometimes compact filter. Is it too late to change naming of the flags? I think in the code it's ok because devs can make the connection. But for users I may be confusing. Maybe it should be `-peerblockfilter` to match `-blockfilterindex`?
10:09 < jnewbery> fjahr: definitely not too late, and definitely a good question to ask. Once things are merged, we're stuck with bad names forever, so we should get it right first time
10:10 < jnewbery> -peerblockfilter seems reasonable. I'll think about it some more and maybe update the flag name
10:11 < fjahr> sounds good!
10:11 < jnewbery> Something that I wondered about as I was reviewing, was the synchronization across threads. Is that something that anyone here thought about?
10:12 < jnewbery> There are two threads to think about. The message hander thread (https://github.com/bitcoin/bitcoin/blob/f54753293fe7355e4280944d766f22054b560ba1/src/net.cpp#L2017). This is the thread that reads messages from the receive buffer and processes them for each peer.
10:12 < jonatack> jnewbery: i'm personally undecided whether to review this PR or not (and for the same reason did not do so for the review club last fall). mind giving an elevator pitch for this PR?
10:13 < jonatack> if this is the wrong moment, not a problem ;)
10:13 < jnewbery> When we receive a GETCFCHECKPT message from a peer, the message handler thread is the one which is calling into ProcessMessages() and doing the work to serve the CFCHECKPT message
10:14 < jonatack> (because if the why isn't clear, the how doesn't matter, and the PR description does not address the why)
10:15 < jnewbery> The other thread we have to think about is the scheduler thread (https://github.com/bitcoin/bitcoin/blob/f54753293fe7355e4280944d766f22054b560ba1/src/init.cpp#L1312), which services async callback in the validationinterface (https://github.com/bitcoin/bitcoin/blob/f54753293fe7355e4280944d766f22054b560ba1/src/validationinterface.h#L84-L161)
10:15 < nehan> do all messages use the message handler thread? how is that handled across messages?
10:16 < fjahr> jonatack: Is it about block filters in general or this PR in particular? i.e. are you more referring to #18876 or #18877?
10:16 < nehan> i guess i'm asking if there's some guidelines on "don't tie up these threads ever"
10:16 < jnewbery> when we receive a block and it's connected, validation will send a BlockConnected signal
10:17 < jnewbery> the scheduler thread will later service that by calling the BlockConnected callback on all the components that subscribed. Here's where the indexer's callback is: https://github.com/bitcoin/bitcoin/blob/f54753293fe7355e4280944d766f22054b560ba1/src/index/base.cpp#L191
10:17 < jnewbery> nehan: yes, all messages from the network are handled by the message handler thread
10:18 < jnewbery> nehan: that's a good question. I haven't seen that guideline anywhere, but it makes sense, right? We shouldn't block the message handler thread, since it's what's doing all the important work
10:18 < theStack> am i right in the assumption that GETCFCHECKPT is just a convenience feature and the same could be achieved by request multiple single GETCFHEADERS?
10:18 < theStack> *requesting
10:18 < nehan> jnewebery: let me clarify my question which was phrased poorly. where is the bulk of the "work" for a message usually handled? is the idea spawn a thread to complete the work for this message, or is it add a bunch of things to different queues that other threads will handle?
10:18 < theStack> (needing much more single network messages and communication rounds of course)
10:18 < jnewbery> jonatack: good question about motivation. I'll get to it in a bit :)
10:19 < sipa> nehan: message is received from network by network thread, put on the received messages queue, where it is processed in the message handler thread
10:19 < jnewbery> nehan: no, the bulk of the work is done by the message handler thread. The only place we farm out work to other threads is script checking for blocks
10:20 < jnewbery> message processing in bitcoin core is almost entirely single-threaded
10:20 < sipa> (for block validation there are separate but fixed script verification threads, which verify all the scripts in parallel for the whole block; for all other things the bulk of the work is in the message handler thread)
10:20 < nehan> sipa: jnewbery: thanks! so to make sure i'm clear: messages are never processed in parallel, because there is only one message handler thread?
10:21 < jnewbery> nehan: correct!
10:21 < sipa> correct; and lots of assumptions (in Core, but also other software) depend on serial processing of messages
10:21 < nehan> jnewbery: sipa: cool. helpful invariant to know
10:21 < jnewbery> there's some low-level preprocessing done by the socket thread, eg we verify the checksum in the socket thread, but all application-level processing is in the message handler thread
10:21 < sipa> there used to be this approach of sending a ping after sending another message to know whether the previous one was finished processing or not
10:22 < jnewbery> someday it'd be nice to have separate threads for net_processing and validation, but we're not there yet
10:22 < sipa> by some software
10:22 < jnewbery> also that assumption is all over our functional tests. We synchronize messages with a ping
10:23 < jnewbery> theStack: The nice thing about the cfcheckpts is that because the headers form a hash chain, you can get your checkpts from somewhere trusted, and then fetch the other headers trustlessly and verify that they connect
10:25 < jonatack> fjahr: yes, the larger direction, beyond being an improvement over bip37
10:25 < jnewbery> you could get any cfheaders from a trusted source, but having the checkpts as a standard in the protocol is nice, I think
10:26 < theStack> jnewbery: oh, i just see now that there is actually a difference between "filter header" and "filter hash"; and that CFHEADERS includes more than just headers
10:26 < jnewbery> they're close enough (1000 blocks), that you can fill in the other headers with a single cfheaders request, and since it's a standard sequence of block heights, the cfcheckpt can be cached and delivered from anywhere
10:28 < jnewbery> theStack: yes, the definition of the header is here: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki#filter-headers. It's the hash of the filter and previous header.
10:28 < jnewbery> so the cfheaders message just provides the first header, and then the hashes of the subsequent filters, so you can reconstuct the headers yourself (and verify that they connect to your checkpoints)
10:29 < jnewbery> ok, back to synchronization. My two questions when thinking about this were: is there the possibility of a data race, and is there the possibility of the message handler thread being blocked?
10:29 < jnewbery> anyone have thoughts about those?
10:31 < fjahr> I did not think about this before but maybe during IBD?
10:32 < jnewbery> fjahr: what about IBD?
10:32 < fjahr> could it be blocked if we get a request during ibd?
10:32 < theStack> fjahr: hmm wouldn't the call to BlockRequestAllowed() just return false if the requested block is not existing yet on the node?
10:33 < theStack> (or maybe before, on LookupBlockIndex)
10:34 < jnewbery> fjahr: perhaps. IBD is when the most messages are going through the validation interface.
10:34 < theStack> s/block/stop block/
10:35 < jnewbery> but I haven't done any testing. The indexer's BlockConnected callback seems to return pretty quickly, so perhaps it isn't a concern
10:35 < jnewbery> theStack: but you could ask for a cfilter for an early block during IBD
10:36 -!- grunch__ [~grunch@2800:810:547:c46:25e8:a7de:f130:fe8] has quit [Remote host closed the connection]     
10:36 < jnewbery> perhaps we should just not serve filters if we're not out of IBD?
10:38 < theStack> what exactly defines the state of "during IBD"? isn't it semantically pretty much the same, just with an (indefinitely ;-)) smaller amount of time between fetching blocks?
10:38 < sipa> during IBD means "while IsInitialBlockDownload() returns true"
10:38 < jnewbery> For the data race concern, you should see that the message handler thread is only reading from leveldb (https://github.com/jnewbery/bitcoin/blob/2a15f9943c065547e896aa221bcf26e7db8bd318/src/index/blockfilterindex.cpp#L390). The schduler thread writes to that database.
10:39 < sipa> it's a heuristic, but it affects lots of things in Bitcoin Core's behavior
10:39 < jnewbery> leveldb's own locking should stop any data races (although I haven't actually looked into that)
10:39 < michaelfolkson> Sorry. Basic question.... Do we forward on transactions/blocks that we know about to other peers while we are still in IBD?
10:40 < jnewbery> michaelfolkson: good question! I think the answer is no, but I'd need to double check that that's always true
10:41 < sipa> jnewbery: https://github.com/google/leveldb/blob/master/doc/index.md#concurrency
10:41 < theStack> sipa: ah, good to know. i didn't expect that
10:41 < jnewbery> thanks sipa!
10:42 < jnewbery> the next PR in the sequence adds an in-memory cache, so if we have multiple threads reading/writing that, we'd need our own locking. See https://github.com/bitcoin/bitcoin/pull/18876/commits/4f159aa8e29c059586bfea51b99074c2b76d196e
10:42 < theStack> because i always saw it as permanent IBD with just extended time in-between blocks :)
10:43 < jnewbery> theStack: no! IBD is very different. It's an in-memory latch. We start off in IBD, when we are at the tip we consider that we're no longer in IBD, and then we never re-enter IBD (unless you restart the node)
10:44 < michaelfolkson> If the answer to transactions/blocks is no then I would guess serving filters during IBD should also be no
10:44 < sipa> i think the most important impact is that during IBD we prefer faster synchronization over partition resistance (we'll kick peers that are too slow to feed us blocks)
10:45 < jnewbery> jonatack: the motivation is that BIP 157 is a better mechanism for light clients, including LN clients, than other protocols
10:45 < theStack> jnewbery: so if a node has finished IBD, the network connections gets -- for whatever reason -- lost for a long time, then the catching up of blocks has a different behaviour?
10:46 < jnewbery> theStack: that is correct
10:47 < sipa> unless you're talking about days... weeks... it doesn't matter much
10:48 < theStack> jnewbery: sipa: okay, that's very interesting. i saw that there is a function for IBD state but assumed it was only used as information for the user and otherwise didn't influence the behaviour
10:48 < sipa> no, i believe it's actually not even exposed to the user
10:48 < jnewbery> I think it's in getblockchaininfo?
10:48 < sipa> ah yes, it's there
10:49 < sipa> added in 0.16
10:49 -!- mol [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
10:49 < jnewbery> (mostly for testing and developers)
10:50 < jnewbery> 10 minutes!
10:50 < jonatack> jnewbery: (heh i began reviewing) right, it is much better than BIP 37, I'm not yet sure if that is a good thing(tm) yet.
10:50 < jnewbery> I have a call at 2, so I'll need to get going promptly
10:51 < jonatack> not to bikeshed on that here; i'll have to work that out for myself
10:51 < jnewbery> I think a good standard for thinking about these things is:
10:52 < jnewbery> 1. Does it have a clear use case (I think in this case, the answer is yes because lightning people want it)
10:52 < jonatack> (HW people too, it seems?)
10:52 < jnewbery> 2. Does it impost a significant or unreasonable resource burden on our software (I think the answer is no here)
10:53 < jnewbery> 3. Is the code correct and maintainable (that's what I need your help for)
10:53 < theStack> just a last note to the IBD topic: i'd agree then to not serve filters during IBD -- not even block headers are served during IBD as i just saw
10:53 < sipa> I think there is a 0. Does this belong in the P2P protocol, or can it be done better in another way - which I think is where some of the controversy is
10:53 < jnewbery> From a software maintainability standpoint, Jimpo's implementation is far better than a lot of the other legacy functionality we have to support
10:54 < michaelfolkson> You mean outside of Core sipa?
10:54 < jonatack> jnewbery: agree, am nevertheless zooming out further: should we be doing this, and if yes, this way?
10:54 < jonatack> sipa: yes
10:55 < sipa> michaelfolkson: no; i mean in the P2P protocol at all, by whatever software
10:55 < michaelfolkson> Ah ok thanks
10:55 < sipa> (of course, we don't control what other people do in their software - but clearly we shouldn't add things that we don't believe belong there)
10:55 < fjahr> jonatack: there is a lot more detail to it but in short I think that if people want to not run a node and don't care about privacy too much then I rather have them run a light client than they keep their money on sth. like coinbase. No doing BIP157 would not make more people run a full node.
10:56 < jnewbery> sipa: I think as long as we satisfy (2) and (3), then (0) doesn't matter so much. Or maybe I should say, 'belongs' in (0) is very subjective. (2) and (3) are also subjective, but not as much.
10:56 < fjahr> To me it is providing an alternative that hopefully will attract more people for who it is an improvement, not a 'downgrade' :)
10:57 < jonatack> yes. i have to work that out. anyway, am reviewing. thanks fjahr jnewbery sipa.
10:57 < sipa> jnewbery: i disagree, but let's keep it at that
10:58 < jonatack> jnewbery: do you want to add this log to the website?
10:58 < jnewbery> ok, I've gotta go. Thanks everyone for coming. I hope you found it useful!
10:59 < jonatack> i can maybe append it to the previous bip157 session page
10:59 < michaelfolkson> Yup very interesting, thanks jnewbery
10:59 < jnewbery> If the first one gets merged, then perhaps we can do another meeting for the next PR in the sequence
10:59 < fjahr> thanks jnewbery, will try to think a little more about threads
10:59 < theStack> thanks for hosting jnerbery! will there be another bip157 review meeting next thursday?
10:59 < theStack> *jnewbery
10:59 < michaelfolkson> haha
10:59 < jonatack> thanks everyone, this is helpful
11:00 < jnewbery> theStack: only if 18877 is merged. No point in reviewing further ahead
11:00 < jnewbery> ok, really going now. Bye!
11:00 < jnewbery> #endmeeting
11:00  * jonatack waves
11:01 < jonatack> jnewbery: will PR adding the log to https://bitcoincore.reviews/16442
11:26 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:358b:f356:e6b1:9e1b] has quit [Ping timeout: 240 seconds]     
11:33 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:b924:7fc5:4d11:6eb2] has joined #bitcoin-core-pr-reviews     
11:41 -!- nothingmuch [~nothingmu@unaffiliated/nothingmuch] has quit [Quit: ZNC - http://znc.in]     
11:41 -!- theStack [~honeybadg@vps1648322.vs.webtropia-customer.com] has quit [Quit: Lost terminal]     
11:48 -!- davterra [~dulyNoded@2601:603:4f00:63d0:a00:27ff:fed5:b769] has quit [Quit: Leaving]     
12:03 < jnewbery> Thanks jon. Sorry - had back to back calls
12:13 -!- Talkless [~Talkless@hst-227-49.splius.lt] has quit [Quit: Konversation terminated!]     
12:46 -!- michaelf_ [~textual@2a00:23c5:be01:b201:cd1d:1bfd:7620:fa2] has joined #bitcoin-core-pr-reviews     
12:49 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:b924:7fc5:4d11:6eb2] has quit [Ping timeout: 240 seconds]     
12:59 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:f48a:5010:9124:3934] has joined #bitcoin-core-pr-reviews     
13:01 -!- michaelf_ [~textual@2a00:23c5:be01:b201:cd1d:1bfd:7620:fa2] has quit [Ping timeout: 244 seconds]     
13:07 -!- seven_ [~seven@2a00:ee2:410c:1300:2cfa:38c8:2d07:d9d5] has quit [Read error: Connection reset by peer]     
13:45 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Remote host closed the connection]     
13:49 -!- thomasb06 [~user@eth-west-pareq2-46-193-0-224.wb.wifirst.net] has quit [Ping timeout: 260 seconds]     
13:55 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews     
13:55 -!- thomasb06 [~user@eth-west-pareq2-46-193-0-224.wb.wifirst.net] has joined #bitcoin-core-pr-reviews     
13:57 < thomasb06> sipa: maybe for big data, the access isn't monothread? (I was reading the meeting discussion)
13:58 -!- lightlike [~lightlike@p200300C7EF1D8900B0A411C1CCBBCDEB.dip0.t-ipconnect.de] has left #bitcoin-core-pr-reviews ["Leaving"]     
13:59 < sipa> i have no idea what you're talking about
14:00 -!- michaelfolkson [~textual@2a00:23c5:be01:b201:f48a:5010:9124:3934] has quit [Quit: Sleep mode]     
14:02 < thomasb06> apparently, data bases are accessed only with one thread?
14:02 < sipa> no     
14:03 < thomasb06> then I misunderstood the conversation
14:03 < sipa> if you're referring to LevelDB, it supports multithreaded access
14:03 < thomasb06> yep, I was talking about it
14:04 < thomasb06> my connexion falls down sometimes, did my compilation message appear in the channel?
14:05 < sipa> i don't think so
14:06 < thomasb06> then here it is (jonatack):
14:06 < thomasb06> -----
14:06 < thomasb06> init.cpp:1879:1: fatal error: error writing to /tmp/ccS1RrVZ.s: No space left on device
14:06 < thomasb06>  1879 | }
14:06 < thomasb06>       | ^
14:06 < thomasb06> compilation terminated.
14:06 < thomasb06> make[2]: *** [Makefile:10684: libbitcoin_server_a-init.o] Error 1
14:06 < thomasb06> make[2]: *** Waiting for unfinished jobs....
14:06 < thomasb06> rpc/net.cpp:787:1: fatal error: error writing to /tmp/ccwf8pzJ.s: No space left on device
14:06 < thomasb06>   787 | }
14:06 < thomasb06>       | ^
14:06 < thomasb06> compilation terminated.
14:06 < thomasb06> make[2]: *** [Makefile:10950: rpc/libbitcoin_server_a-net.o] Error 1
14:06 < thomasb06> rpc/blockchain.cpp:2391:34: fatal error: error writing to /tmp/ccpgRQJf.s: No space left on device
14:06 < thomasb06>  2391 | NodeContext* g_rpc_node = nullptr;
14:06 < thomasb06>       |                                  ^
14:06 < sipa> thomasb06: lesson in IRC courtesy, don't paste things of more than 3 lines
14:06 < thomasb06> compilation terminated.
14:06 < thomasb06> make[2]: *** [Makefile:10908: rpc/libbitcoin_server_a-blockchain.o] Error 1
14:06 < sipa> use a pastebin site
14:06 < thomasb06> rpc/rawtransaction.cpp:1846:1: fatal error: error writing to /tmp/cczjIrIz.s: No space left on device
14:07 < thomasb06>  1846 | }
14:07 < thomasb06>       | ^
14:07 < thomasb06> compilation terminated.
14:07 < thomasb06> make[2]: *** [Makefile:10964: rpc/libbitcoin_server_a-rawtransaction.o] Error 1
14:07 < thomasb06> make[2]: Leaving directory '/home/user/github/bitcoin/src'
14:07 < thomasb06> make[1]: *** [Makefile:17329: all-recursive] Error 1
14:07 < thomasb06> make[1]: Leaving directory '/home/user/github/bitcoin/src'
14:07 < thomasb06> make: *** [Makefile:780: all-recursive] Error 1
14:07 < thomasb06>       
14:07 < thomasb06> -----
14:07 < thomasb06> aie... sorry
14:07 < sipa> your disk seems full
14:08 < sipa> or at least your tmp filesystem is
14:08 < thomasb06> xdiskusage does't seem to show a full /tmp
14:08 < sipa> how much free space does it have?
14:09 < thomasb06> in the bar, 14G in /
14:10 < thomasb06> (and 29G in ~ but it has nothing to do)
14:11 < thomasb06> also, there is no file /tmp/ccS1RrVZ.s
14:11 < sipa> sure, it's a temporary build artefact
14:11 < sipa> it's deleted when cleaning up
14:12 < thomasb06> at least this normal
14:12 < sipa> 14G does not seem too little
14:14 < thomasb06> since I only build the code, for having asked several times, it should be enough
14:19 < thomasb06> here is the trace: https://x0.at/1f-
15:01 -!- thomasb06 [~user@eth-west-pareq2-46-193-0-224.wb.wifirst.net] has quit [Remote host closed the connection]     
15:07 -!- seven_ [~seven@2a00:ee2:410c:1300:f1f9:7f46:d994:e48f] has joined #bitcoin-core-pr-reviews     
15:47 -!- brakmic [~brakmic@185.183.85.108] has quit []     
16:14 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]     
16:21 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews     
16:51 -!- jonatack_ [~jon@37.165.219.151] has joined #bitcoin-core-pr-reviews     
16:54 -!- jonatack [~jon@37.164.233.66] has quit [Ping timeout: 260 seconds]     
17:01 -!- seven_ [~seven@2a00:ee2:410c:1300:f1f9:7f46:d994:e48f] has quit [Remote host closed the connection]     
17:01 -!- seven_ [~seven@2a00:ee2:410c:1300:f1f9:7f46:d994:e48f] has joined #bitcoin-core-pr-reviews     
17:11 -!- jonatack_ [~jon@37.165.219.151] has quit [Quit: jonatack_]     
17:11 -!- jonatack [~jon@213.152.180.5] has joined #bitcoin-core-pr-reviews     
17:35 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
17:36 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 256 seconds]     
17:58 -!- jonatack [~jon@213.152.180.5] has quit [Ping timeout: 246 seconds]     
18:04 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has quit [Ping timeout: 260 seconds]     
19:25 -!- mol_ [~mol@unaffiliated/molly] has quit [Ping timeout: 246 seconds]     
19:50 -!- mol [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
20:30 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 272 seconds]     
21:19 -!- vasild_ [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews     
21:22 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]     
21:22 -!- vasild_ is now known as vasild     
21:54 -!- dfmb___ [~dfmb_@unaffiliated/dfmb/x-4009105] has joined #bitcoin-core-pr-reviews     
22:16 -!- Talkless [~Talkless@hst-227-49.splius.lt] has joined #bitcoin-core-pr-reviews     
22:29 -!- Talkless [~Talkless@hst-227-49.splius.lt] has quit [Quit: Konversation terminated!]     
22:30 -!- Talkless [~Talkless@hst-227-49.splius.lt] has joined #bitcoin-core-pr-reviews     
22:32 -!- mol [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
22:58 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
23:12 -!- brakmic [~brakmic@185.183.85.108] has joined #bitcoin-core-pr-reviews     
23:26 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
23:29 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 256 seconds]     
23:30 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Ping timeout: 240 seconds]     
23:30 -!- seven_ [~seven@2a00:ee2:410c:1300:f1f9:7f46:d994:e48f] has quit [Read error: Connection reset by peer]     
23:32 -!- mol_ [~mol@unaffiliated/molly] has quit [Ping timeout: 264 seconds]     
23:34 -!- mol [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
23:34 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-core-pr-reviews     
23:47 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has quit [Ping timeout: 264 seconds]     
23:47 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews     
23:50 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 272 seconds]     
23:54 -!- jesseposner [~jesseposn@c-67-188-220-154.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews