--- Log opened Wed Apr 27 00:00:07 2022
00:03 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@ip72-194-104-106.oc.oc.cox.net] has joined #bitcoin-core-pr-reviews
00:22 -!- belcher [~belcher@user/belcher] has quit [Ping timeout: 276 seconds]
00:22 -!- belcher [~belcher@user/belcher] has joined #bitcoin-core-pr-reviews
00:42 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
00:47 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Ping timeout: 240 seconds]
01:05 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@ip72-194-104-106.oc.oc.cox.net] has quit [Ping timeout: 256 seconds]
01:16 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
01:18 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@ip72-194-104-106.oc.oc.cox.net] has joined #bitcoin-core-pr-reviews
01:20 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Ping timeout: 240 seconds]
01:27 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@ip72-194-104-106.oc.oc.cox.net] has quit [Ping timeout: 240 seconds]
01:48 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
01:49 -!- evanlinjin__ [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
01:53 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Ping timeout: 260 seconds]
01:58 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:adf2:32d7:1c5a:583f] has joined #bitcoin-core-pr-reviews
02:01 -!- evanlinjin__ [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
02:37 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
02:41 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Ping timeout: 240 seconds]
03:02 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:adf2:32d7:1c5a:583f] has quit [Ping timeout: 250 seconds]
03:21 -!- lifeaswesknowit [~shoeless@c-67-176-12-139.hsd1.co.comcast.net] has quit [Ping timeout: 250 seconds]
03:26 -!- brunoerg [~brunoerg@187.183.43.40] has joined #bitcoin-core-pr-reviews
03:31 -!- brunoerg [~brunoerg@187.183.43.40] has quit [Ping timeout: 276 seconds]
03:31 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:adf2:32d7:1c5a:583f] has joined #bitcoin-core-pr-reviews
03:37 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:adf2:32d7:1c5a:583f] has quit [Ping timeout: 250 seconds]
03:46 -!- cryptoquick8 [~cryptoqui@8.46.89.33] has joined #bitcoin-core-pr-reviews
03:46 -!- cryptoquick [~cryptoqui@8.46.89.33] has quit [Ping timeout: 260 seconds]
03:46 -!- cryptoquick8 is now known as cryptoquick
03:58 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:adf2:32d7:1c5a:583f] has joined #bitcoin-core-pr-reviews
04:00 -!- brunoerg [~brunoerg@187.183.43.40] has joined #bitcoin-core-pr-reviews
04:04 -!- brunoerg [~brunoerg@187.183.43.40] has quit [Ping timeout: 240 seconds]
04:49 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
04:54 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Ping timeout: 240 seconds]
05:02 -!- evanlinjin__ [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
05:04 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:adf2:32d7:1c5a:583f] has quit [Ping timeout: 248 seconds]
05:05 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
05:08 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
05:10 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Remote host closed the connection]
05:10 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
05:25 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Remote host closed the connection]
05:27 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
05:31 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:adf2:32d7:1c5a:583f] has joined #bitcoin-core-pr-reviews
05:37 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:adf2:32d7:1c5a:583f] has quit [Ping timeout: 250 seconds]
05:50 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@ip72-194-104-106.oc.oc.cox.net] has joined #bitcoin-core-pr-reviews
05:57 -!- a1ph4byte [~a1ph4byte@2600:1700:22f0:a7c0:bc46:d63f:2e5f:7485] has joined #bitcoin-core-pr-reviews
06:14 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
06:33 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
06:37 -!- evanlinjin_ [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
06:39 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
06:41 -!- evanlinjin_ [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Client Quit]
06:42 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
06:53 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@ip72-194-104-106.oc.oc.cox.net] has quit [Ping timeout: 250 seconds]
07:21 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
07:25 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Ping timeout: 240 seconds]
07:26 -!- lifeaswesknowit [~shoeless@c-71-237-122-116.hsd1.co.comcast.net] has joined #bitcoin-core-pr-reviews
07:38 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
08:41 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Ping timeout: 250 seconds]
09:01 -!- b_1o1 [~b_1o1@189.236.26.138] has joined #bitcoin-core-pr-reviews
09:17 -!- b_1o1 [~b_1o1@189.236.26.138] has quit [Quit: Connection closed]
09:20 -!- Talkless [~Talkless@mail.dargis.net] has joined #bitcoin-core-pr-reviews
09:30 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
09:34 -!- Frank0 [~Frank0@186.84.21.96] has joined #bitcoin-core-pr-reviews
09:54 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Remote host closed the connection]
09:54 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
09:57 -!- b_1o1 [~b_1o1@189.236.26.138] has joined #bitcoin-core-pr-reviews
09:57 -!- jacobpfickes [~jacobpfic@104.165.250.208] has joined #bitcoin-core-pr-reviews
09:57 -!- svav [~svav@82-69-86-143.dsl.in-addr.zen.co.uk] has joined #bitcoin-core-pr-reviews
09:57 -!- ccdle12 [~ccdle12@243.222.90.149.rev.vodafone.pt] has joined #bitcoin-core-pr-reviews
09:58 -!- randomsats [~randomsat@cpe1033bfa9e0e7-cm1033bfa9e0e5.cpe.net.cable.rogers.com] has joined #bitcoin-core-pr-reviews
10:00 < glozow> #startmeeting
10:00 < glozow> hi everyone!
10:00 < svav> Hi
10:00 < a1ph4byte> Hello!
10:00 < emzy> hi
10:00 < Frank0> hi
10:00 -!- b_1o1 [~b_1o1@189.236.26.138] has quit [Client Quit]
10:00 < theStack> hi!
10:00 < glozow> Welcome to PR Review Club! Anyone here for the first time?
10:00 -!- b_1o1 [~b_1o1@189.236.26.138] has joined #bitcoin-core-pr-reviews
10:00 < sipa> hi
10:00 < lightlike> hi
10:00 < b10c> hi
10:01 < glozow> we're looking at #24644, "Add tracepoints and algorithm information to coin selection" today
10:01 < Frank0> yes me
10:01 < b_1o1> hi all
10:01 < a1ph4byte> First Timer here
10:01 < glozow> Notes in the usual place: https://bitcoincore.reviews/24644
10:01 < jacobpfickes> Hi all!
10:01 -!- effexzi [uid474242@id-474242.ilkley.irccloud.com] has joined #bitcoin-core-pr-reviews
10:01 < ccdle12> hi
10:01 < Frank0> first time
10:01 < schmidty> hi
10:01 < glozow> Awesome, welcome Frank0 and a1ph4byte!
10:01 < effexzi> Hi every1 
10:01 < svav> To the new people, where did you here about this meeting if you don't mind sharing? Thanks
10:01 < glozow> Did anybody get a chance to review the PR or look at the notes? How about a y/n
10:02 < a1ph4byte> browsing the bitcon-core contributor notes
10:02 < Frank0> chain code labs bitcoin seminar
10:02 < emzy> y (a little)
10:02 < a1ph4byte> n
10:02 < svav> y looked at the notes
10:02 < b10c> y
10:02 < b_1o1> y
10:02 < svav> Thanks Frank0
10:02 < larryruane> hi
10:03 < theStack> n
10:03 < glozow> could somebody summarize for us what this PR does?
10:04 < svav> It adds tracepoints into the wallet's coin selection code, so we can better understand how it's performing
10:04 < glozow> svav: perfect, thank you!
10:04 < svav> ... and how "good" coin selection is
10:05 < glozow> And can somebody quickly summarize for us what tracepoints are?
10:05 < larryruane> a way of getting access to internal state of a running bitcoind for debugging or general understanding
10:06 < larryruane> it's sort of like logging (where you can enable particular categories) but less intrusive (?)
10:06 < svav> Once a tracepoint is reached, it can pass data about process internals to a userspace script for further processing. This is great for observability and allows for debugging, testing, and monitoring.
10:06 < glozow> larryruane: right! And what's special about these tracepoints in particular? Is there a difference between tracepoints and logs?
10:07 < larryruane> when you say these, do you mean coin selection tracepoints?
10:07 < glozow> svav: 👍
10:07 < glozow> I mean, USDTs versus other types of telemetry
10:07 < svav> A tracepoint is triggered by particular code activation
10:07 < svav> So a tracepoint shows what code is being called, unlike a log
10:08 < a1ph4byte> May I ask, what is meant by "coin selection"
10:08 < larryruane> The other difference I think of with tracepoints is that, with logging, you can later process the log file (to summarize what's in it), but there could be a huge amount of disk space consumed ... with tracepoints, you can sort of "compress" the information on the fly
10:08 < glozow> for anyone that's unfamiliar with USDTs, https://github.com/bitcoin/bitcoin/pull/22006 is a good place to start
10:08 -!- will [~will@187.34.151.79] has joined #bitcoin-core-pr-reviews
10:08 < sipa> svav: What do you mean by "what code is being called" ?
10:09 < b10c> logging is primarily a interface for humans, tracepoints are a interface for machines 
10:09 < svav> a1ph4byte Coin Selection - The process of selecting UTXOs (“coins”) from a wallet’s UTXO pool in order to fund a transaction’s payment(s).
10:09 -!- PaperSword1 [~PaperSwor@50.126.96.22] has joined #bitcoin-core-pr-reviews
10:10 -!- PaperSword [~PaperSwor@50.126.96.22] has joined #bitcoin-core-pr-reviews
10:10 < b10c> we can still parse log messages, but the contents might change over time and we might end up e.g. printing a hash as hex and then parsing it back in which isn't efficient
10:11 -!- PaperSword1 [~PaperSwor@50.126.96.22] has quit [Client Quit]
10:11 < Murch> a1ph4byte: A general term for the strategies and algorithms used to pick the inputs for transactions
10:11 < sipa> I think another important difference is that logging is an actual action, where tracepoints are just hooks that an external process can plug into. A tracepoint on itself does nothing unless something uses it.
10:11 -!- pop [~pop@c-71-231-107-61.hsd1.wa.comcast.net] has joined #bitcoin-core-pr-reviews
10:11 < glozow> thanks sipa and b10c!
10:11 < larryruane> i have a really basic question, when a thread hits an active tracepoint, does the thread suspend until the data is received by the tracing script? Or is there a memory queue of tracing events so the thread can continue asynchronously?
10:12 < PaperSword> A tracepoint can still execute functions within the TRACEx call.
10:12 < sipa> stijnbtc: Note that people outside of matrix can't see your emoji response.
10:12 < stijnbtc[m]1> Ah good to know! 
10:12 < svav> sipa What I mean is a tracepoint can provide specific information about what code is doing ... at specific points in the code ...
10:13 < glozow> Feel free to continue a background thread for general USDT questions. I'll also start with the questions about the PR.
10:13 < glozow> What 4 tracepoints does this PR add, and what information do they collect?
10:13 < PaperSword> No suspension I believe, data goes into the ringbuffer where it can be over written if not read right awway.
10:13 < PaperSword> sorry
10:15 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
10:15 < svav> For reference, I think SystemTap is being used to provide the tracepoints https://sourceware.org/systemtap/
10:15 < pop> For reference, here is the doc/tracing.md to see a generalized tracepoint diagram: https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md
10:16 < b10c> larryruane: in assembly, the tracepoint is a literal NOP (no operation). If we tell the kernel to hook into the tracepoint, it executes a small eBPF bytecode program e.g. adding data to a eBPF map where it can be read asynchronously from a userspace program
10:16 < b10c> It doesn't suspend
10:16 < svav> So this PR adds 4 new tracepoints:
10:16 < svav> After SelectCoins returns in order to observe the SelectionResult
10:16 < svav> After the first CreateTransactionInternal to observe the created transaction
10:16 < svav> Prior to the second CreateTransactionInternal to notify that the optimistic avoid partial spends selection is occurring
10:16 < svav> After the second CreateTransactionInternal to observe the created transaction and inform which solution is being used.
10:17 < glozow> svav: well prepared :)
10:18 < PaperSword> coin_selection:selected_coins collects 1. Wallet name as `pointer to C-style string` 
10:18 < PaperSword> 2. Coin selection algorithm name as `pointer to C-style string` 
10:18 < PaperSword> 3. Selection target value as `int64`
10:18 < PaperSword>  4. Calculated waste metric of the solution as `int64`
10:18 < PaperSword> 5. Total value of the selected inputs as `int64`
10:18 < Murch> glozow: The tracepoints collect information on the algorithms that produced the input selection, the total amount selected, the waste score, some details on fees, change output position, and whether the solution is `avoid_partial_unspents` compliant
10:18 < sipa> b10c: But execution of bitcoind is suspect while the kernel executes the eBPF program - it's the reading out of the results that is done asynchronously?
10:18 < sipa> s/suspect/suspended/
10:18 < sipa> (I've never actually used eBPF/USDT)
10:18 < glozow> Murch: PaperSword: nice
10:19 < glozow> so svav told us what the 4 tracepoints are, and Murch and PaperSword told us what information is being collected
10:19 < glozow> Why might we be interested in the algorithm and waste score? How might we use this information to improve coin selection?
10:19 < larryruane> b10c: so the NOP is replaced at runtime (after the program is loaded in memory)? Sort of the way breakpoints work?
10:19 < b10c> sipa: yes, while the eBPF program runs bitcoind is suspended. So hooking into tracepoints in tight loops might affect performance
10:20 < PaperSword> There is more info collected that was just the the args passed for 1/4 traces
10:20 < sipa> Got it.
10:20 < pop> Improving the coin selection algorithm has obvious downstream benefits for all wallets that are based on/rely on the bitcoind wallet.
10:20 < glozow> PaperSword: right, that's the info for `selected_coins` tracepoint yes?
10:20 < PaperSword> Trace is only optimized to just a NOP if compiled without ePBF support?
10:20 -!- Observer83 [~Observer@cpe-23-242-148-67.socal.res.rr.com] has joined #bitcoin-core-pr-reviews
10:21 < PaperSword> yes that is correct.
10:21 < pop> Is it relevant to ask how coin selection algorithms were developed prior to the proposal of these 4 tracepoints?
10:21 < glozow> pop: right
10:22 < Murch> While it's easy to evaluate a coin selection algorithm on a single situation (UTXO pool and selection target), the overall problem we're interested in is the emergent behavior of various algorithms over longer scenarios of payment sequences and feerates.
10:22 < b10c> larryruane: yes, I think of it similar to breakpoints in a debugger on the NOP, the positions of these NOPs are written into a ELF note of the bitcoind binary
10:22 -!- drdprtrbrts [~drdprtrbr@154.0.128.44] has joined #bitcoin-core-pr-reviews
10:22 < theStack> PaperSword: i'd assume that if we compile without eBPF support, there are not even NOPs, because the TRACE... defines are replaced by empty strings
10:23 < Murch> The tracepoints allow us to observe how the UTXO pool evolves over time and to assess the overall fee expenditures as well as the individual outcomes of each payment
10:23 < glozow> Murch: yes thank you.
10:23 < achow101> pop: the tracepoints let us do simulations. prior to tracepoints, these simulations required an aditional patch which adds a couple of globals and RPCs that let us measure some things. but these could not be upstreamed so had to be maintained separately
10:23 -!- stijn [~stijn@2a02-a44e-3f48-1-831a-d2c7-df9f-e068.fixed6.kpn.net] has joined #bitcoin-core-pr-reviews
10:23 < pop> Murch: So without net tracepoints to track coinselection metrics over longer periods and map the interactions between multiple algorithms, there is no way to evaluate the emergent behavior of coin selection?
10:24 < glozow> I also think it's a good way to evaluate the accuracy of our waste metric
10:24 < sipa> pop: Sure there is, but you couldn't do it with an unmodified bitcoind. Tracepoints provide a way for profiling software to hook into bitcoind, unmodified.
10:24 < b10c> PaperSword: when compiled without tracing support, there's nothing tracing related in the code. The TRACEx makros are empty if tracing is disabled
10:25 < PaperSword> per https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md  "even if the tracepoint is not used. For example, avoid serialization and parsing."
10:25 < Murch> pop: Previously we had either created separate simulation frameworks or modified a copy of Bitcoin Core to add the corresponding logging. Having the tracepoints allows us to keep the log generation and processing in a separate project which makes it easier to apply the tracing to many different states of the codebase
10:25 < PaperSword> *"Although the tracepoint itself only has overhead when enabled, the code to compute arguments is always run"
10:26 < pop> sip: I see. So the pull in question consolidates patches into the main bitcoind codebase and removes the need to maintain additional code.
10:26 < glozow> right, with tracepoints we get the win-win of upstream support for observability + people who don't care about that won't be imapcted
10:26 < a1ph4byte> Where are the coin selection algorithms implemented? Is this apart of the bitcoin-core or is this implemented by external software apps?
10:26 < sipa> Part of Bitcoin Core.
10:26 < glozow> see src/wallet/coinselection.{h,cpp}
10:26 < Murch> a1ph4byte: The coin selection algorithms are part of the Bitcoin Core codebasez
10:26 < PaperSword> glozow: Correct, though it can lead to messy looking code because each trace takes up space. Just my opinion.
10:27 < b10c> PaperSword: this guidance is relevant for people who run a bitcoind with tracing support, but don't hook into the tracepoints. Release builds have tracepoint support, so we assume that's the case for a majority of our users.
10:27 -!- stijn [~stijn@2a02-a44e-3f48-1-831a-d2c7-df9f-e068.fixed6.kpn.net] has quit [Client Quit]
10:27 < glozow> sure, though a few lines of code is a trivial cost to pay for the observability benefits
10:27 < PaperSword> b10c: Thank you for clarifying.
10:28 < sipa> PaperSword: By "space" you mean lines of code in the source?
10:28 < PaperSword> glozow: I agree. 
10:28 < PaperSword> sipa: yes
10:28 < PaperSword> I can't speak on the waste metric.
10:29 < sipa> Ok, sure, it takes up source code space, but any alternative that doesn't have that will just not have any logging/tracing/observing of the relevant metric at all.
10:29 < theStack> b10c: "Release builds have tracepoints support" oh that's interesting, i would have guessed that they are only useful for developers and are disabled for releases
10:29 < glozow> for those interested, here's `AttemptSelection` which tries various algorithms: https://github.com/bitcoin/bitcoin/blob/f0a834e2f10a0aa60c7cc76e9f3eb090168a32e5/src/wallet/spend.cpp#L379
10:29 < larryruane> so do we ask important bitcoind users, such as exchanges, to enable tracepoints to learn about real-world behavior? (and send us results) Or is the intention that tracepoints are only for us dev types?
10:29 -!- azor [~azor@200.109.50.144] has joined #bitcoin-core-pr-reviews
10:29 < glozow> here is where the waste metric is defined: https://github.com/bitcoin/bitcoin/blob/f0a834e2f10a0aa60c7cc76e9f3eb090168a32e5/src/wallet/coinselection.h#L225
10:29 < PaperSword> correct me if I am wrong release builds only have trace support when built with 'depends'
10:29 -!- potatowhale [~potatowha@4.53.92.114] has joined #bitcoin-core-pr-reviews
10:30 < glozow> and here are the 3 coin selection algorithms we use https://github.com/bitcoin/bitcoin/blob/f0a834e2f10a0aa60c7cc76e9f3eb090168a32e5/src/wallet/coinselection.h#L292-L304
10:30 < sipa> release builds are built with depends
10:30 < sipa> (by "release builds" we mean the binaries that are distributed on bitcoincore.org etc)
10:30 < PaperSword> glozow: I was already able to take a look at the metric, but spent a lot of time trying to get the tests in this PR to pass on RHEL linux. I was unable to successfuly run the tests from this PR.
10:30 < pop> Are there currently only 11 tracepoints implemented as listed in doc/tracing.md?
10:31 < PaperSword> sipa: thanks
10:31 < glozow> larryruane: yeah that would be amazing if people could collect data for us this way
10:31 < achow101> larryruane: the intention was to use it with the simulation script that I've written
10:31 < Murch> We also had a PR Review club on a waste metric related PR https://bitcoincore.reviews/22009
10:31 < PaperSword> I love tracepoints.
10:32 < glozow> seems like achow101 and Murch have been running the simulations on transaction data collected from some donors
10:32 < achow101> I would not really expect others to log with these tracepoints and give us the data. it does not preserve privacy
10:32 < glozow> achow101: what about aggregate stats?
10:33 < glozow> if a really huge enterprise wallet could tell us the distribution of algorithms used at different feerates, for example
10:33 < b10c> theStack: It makes sense to have the tracepoints in release builds to allow people to trace their production setups. Switching binaries is often not something you want to do if your trying to debug a problem. 
10:33 < Murch> larryruane: We currently have three datasets, an online gambling service'¿ payment sequence, a merchant's inbound payments, and another services payments. Still looking for something representative of individual users
10:33 < glozow> although i hope enterprise wallets aren't using bitcoin core wallet
10:34 < achow101> glozow: aggregate data could be useful
10:34 < larryruane> some other projects (storage systems) have this "phone home" idea, but it was always a huge privacy concern ... I can understand why we can't / shouldn't do anything like that!
10:36 < Murch> Yeah, maybe to make it clear, this is not a telemetry function, it's just for users to hook into stuff running on their own computer. Our simulation scenarios are merely lists of the incoming and outgoing amounts they've processed without additional information (and slightly fuzzified amounts for privacy)
10:36 < pop> achow101: it's hard to imagine a dataset of transactions that wouldn't be personally identifiable, especially if those transactions have made it into blocks
10:36 < PaperSword> Note that tracing programs have to be run as root and only support linux right now. 
10:37 < achow101> pop: it's possible to anonymize payment datasets by adding/subtracting a small random amount to each payment amount
10:37 < PaperSword> To hook into eBPF tracepoints is quite a deliberate action. 
10:38 < pop> achow101: but wouldn't that affect the waste metric? If you are using the dataset to evaluate coin selection algorithms?
10:38 < theStack> b10c: that makes sense
10:39 < a1ph4byte> a more general question: what is the rationale behind developing coin selection algorithms as opposed to a simplistic FIFO model?
10:39 < achow101> pop: with the simulation framework, we make each payment and use the tracepoints to observe what coin selection did
10:40 < Murch> pop: it's an amount and a feerate for each payment, e.g. someone paid  
10:40 < Murch> 0.001032 ₿  at a feerate of 0.00015817 ₿/kvB
10:40 < Murch> they might tell us 
10:40 < Murch> 0.00102723,0.00015831
10:40 < Murch> so, we don't know the txid, time, or actual values
10:40 < theStack> could the tracepoints probably also serve as a replacement to the zmq notifications, on the long term? (don't know too much on either of the two areas)
10:41 < achow101> a1ph4byte: we want to have our automatic coin selection behave in a smart way, e.g. reduce fees when feerates are high, consolidate more when feerates are low. a simplistic algorithm can result in unexpected or undesired behavior
10:41 < Murch> pop: We run the data as a benchmark against different coin selection improvements to compare which perform better on the scenario. So the exact amounts aren't that important
10:41 < glozow> a1ph4byte: you might find useful information in these notes https://bitcoincore.reviews/22009. FIFO would be expensive and leak information about the wallet, etc.
10:41 < PaperSword> theStack: IMO this would be amazing but tracepoints have lower compatibility. Linux as root only.
10:42 < achow101> a1ph4byte: there are also privacy considerations, and maintaining a usable utxo pool for the wallet (e.g. not producing sand (near-dust) outputs)
10:42 < Murch> a1ph4byte: I wrote a bit about that here: https://bitcoin.stackexchange.com/a/32445/5406
10:42 < a1ph4byte> glozow super helpful link!
10:42 < pop> achow101: Murch: So the important thing is to simply have data that has a close relationship with actual usage. The specific relationship between payment and feerate isn't critical at this point in coin selection algorithm development?
10:42 < b10c> theStack: the tracepoints are currently Linux only. MacOS and Win might need ZMQ notifications too
10:43 < achow101> pop: yes
10:43 < theStack> PaperSword: indeed, needing to run as root seems to be quite of a drawback (though i think someone mentioned at #bitcoin-core-dev recently that laanwj managed to get them to run without root... maybe someone knows more details)
10:43 < Murch> Right, we just want something that is representative of how people use Bitcoin to reason about whether what we're doing improves their outcome
10:44 < theStack> b10c: good point
10:44 < Murch> They don't even have to be the Bitcoin Core wallet
10:44 < b10c> theStack: currently we only make data available via tracepoints that's already present in the function they are called in. that means, transactions and blocks might need additional serialization which could be "expensive"
10:44 < glozow> It would be nice to simulate what would happen if we lowered the consolidation feerate
10:46 < Murch> glozow: I have a branch for that
10:46 < laanwj> theStack: PaperSword: yes, two capabilities are needed for tracepoints, full root isn't needed (on recent-ish kernels) see https://github.com/bitcoin/bitcoin/pull/24358#issuecomment-1083149220
10:46 < glozow> Murch: nice
10:46 < b10c> theStack: https://github.com/bitcoin/bitcoin/pull/24358#issuecomment-1083149220
10:46 < glozow> Also might be nice to factor "remaining number of confirmed UTXOs in wallet after this transaction is created" in to waste metric
10:46 < theStack> laanwj: b10c: thanks!
10:47 < PaperSword> laanwj: I saw this :D Haven't had a chance to try it yet.
10:47 < PaperSword> laanwj: thanks 
10:48 < Murch> glozow: josibake and I talked about that last week as well
10:49 < Murch> He suggested that we might introduce additional metrics and use scores from all of them to pick the best selectionresult rather than just the wastemetric
10:49 < glozow> endless room for improvement in coin selection
10:49 < Murch> Two that we talked about were "reliability" and "privacy" metrics
10:49 < glozow> Yeah, you might want a "privacy score" encapsulating whether you have a change output, how many inputs you're pulling together of what outputtypes, etc
10:50 < glozow> but hard to quantify, i'm sure
10:50 < Murch> E.g. spending unconfirmed inputs would reduce your reliability score, and mixing inputs with different script types reduce your privacy score
10:50 < glozow> @ review clubbies, we have a few more questions in the notes - do we want to go through them?
10:50 < Murch> Indeed
10:51 < PaperSword> Yes
10:51 < glozow> We don't have to - I think the conversation has been very educational and on topic. Just asking
10:51 < Murch> Yeah, we got a bit sidetracked, go ahead
10:51 < glozow> Okay cool. What is a C-style string and why do the tracepoints pass this type of string?
10:52 < PaperSword> Null terminated string and it's passed because of the fact it's can be used in a ring buffer?
10:53 < pop> glozow: I don't think this one was answered: 4. Instead of instrumenting the coin selection code with tracepoints, why don’t we just add logs?
10:53 < PaperSword> Without termination in a ringbuffer specifically the ePBF program would not know when the var passed ended?
10:53 < glozow> pop: Oh yeah, I skipped that one since we had some discussion about it in the beginning. We can go over that one too
10:55 < PaperSword> Also without termination the user would have to pass a len arg, that would be crazy expensive given the 6/12 arg limit on trace functions
10:55 < Murch> pop: I think I kinda did
10:55 < pop> murch: sorry, you're right
10:55 < glozow> PaperSword: yeah, C-strings don't know their own length so you just need to look for \0 when parsing. I don't know the answer to "why does it need to be a C-string," I assumed we have to use primitive data types or something
10:56 < achow101> the c-style string is because the ebpf program is in C :)
10:56 < glozow> aha there we go
10:56 < achow101> it doesn't compile if you give it the std::string
10:56 < Murch> "Previously we had either created separate simulation frameworks or modified a copy of Bitcoin Core to add the corresponding logging. Having the tracepoints allows us to keep the log generation and processing in a separate project which makes it easier to apply the tracing to many different states of the codebase"
10:56 < b10c> We can't pass C++'s std::string to C
10:56 < PaperSword> my bad,
10:56 < glozow> are `char[]` and `char*` the same thing in C?
10:56 < PaperSword> I was thinking about just sending a char array instead
10:57 < PaperSword> sans termination*
10:57 < sipa> glozow: *almost*
10:58 < sipa> sizeof(char[]) gives the length of the array and &(char[]) returns a pointer to the first element of the array (so, itself). For all other purposes, a char[] just degenerates into a char*
10:58 < sipa> C arrays are bad.
10:58 < glozow> I guess in this scenario, then, they would be equivalent
10:59 < sipa> Yes, you can't pass an array as an argument in C, for example - it degenerates into a pointer.
10:59 < sipa> Also if you define a function that takes a char[] as argument, it's actually an argument of type char*.
10:59 < PaperSword> sipa: correct
11:00 < theStack> sipa: there are fixed size array parameters possible though, isn't it? like "void foo(int bla[5])"... at least i vaguely remember that i used this years ago, maybe it was non-standard though
11:00 < sipa> theStack: Nope, that is exactly equivalent to writing "void foo(int* bla)".
11:00 < sipa> (IIRC)
11:00 < glozow> We're out of time, hopefully this was fun!
11:01 < sipa> You can write it, but it's just information for the programmer.
11:01 < glozow> #endmeeting
11:01 < theStack> sipa: okay, so it is accepted, but the specified size doesn't have any meaning
11:01 < sipa> (going to test that now, theStack)
11:01 < emzy> Thank you glozow and all!
11:01 < PaperSword> thanks
11:01 < pop> thanks, really informative
11:01 < svav> Thanks glozow and all!
11:01 < b10c> thanks!
11:01 < theStack> sipa: at least the compiler could be so nice to use it to detect an obvious out-of-bounds access at compile-time :)
11:01 < a1ph4byte> glozow This was wildly beneficial and surprisingly approachable! Thank you!
11:01  * emzy running functional tests for the PR now.
11:01 < theStack> thanks for hosting glozow!
11:01 < larryruane> thanks glozow!
11:02 < b_1o1> glozow: thanks for hosting, and achow101
11:02 -!- jacobpfickes [~jacobpfic@104.165.250.208] has quit [Quit: Connection closed]
11:02 < Murch> Thanks for hosting and pinging me ^^
11:02 < b10c> i think we could do with a bit more documentation/tutorials around the tracing functionally
11:02 < glozow> Thanks all for coming! Sorry if you were expecting us to stick to the questions, I personally am very happy we didn't. Feel free to ask if you want answers to the questions, I'll be around for a while.
11:02 -!- potatowhale [~potatowha@4.53.92.114] has quit [Quit: Connection closed]
11:02 < sipa> theStack: https://godbolt.org/z/bbvKqaa1b
11:02 -!- azor [~azor@200.109.50.144] has quit [Quit: Connection closed]
11:03 < PaperSword> b10c: the purpose of tracing seems to be quite misunderstood and loosely defined.
11:03 < PaperSword> just my opinion. 
11:03 < sipa> GCC 11.3 even warns you that sizeof(int[5] argument) reports sizeof(int*) actually.
11:04 < theStack> sipa: interesting!
11:04 < sipa> Linus Thorvalds has a rant somewhere that you should never ever pass a C array as an argument, because it's so confusing.
11:04 < pop> Would anyone be willing to break this one down? 6. What’s the difference between the two calls to CreateTransactionInternal? What does it mean to Avoid Partial Spends?
11:05 < sipa> theStack: https://lkml.org/lkml/2015/9/3/428
11:05 -!- Observer83 [~Observer@cpe-23-242-148-67.socal.res.rr.com] has quit [Quit: Connection closed]
11:05 < achow101> pop: the avoid partial spends feature groups together all UTXOs for the same address and treats them as a single UTXO during coin selection. This means that all UTXOs for the same address are all spent at the same time.
11:06 < theStack> sipa: i was hoping that the compiler would at least warn if you'd example access c[5] in func, but apparently it doesn't; so it really treats it just as pointer and that's it
11:06 < achow101> if avoid partial spends is off (it is off by default), we would do a CreateTransactionInternal without it, then do it again with APS on. Then we choose the "better" of the two solutions
11:07 < pop> achow101: Is this because, once you spend a single UTXO from the address you will have revealed the private key for that address, rendering all of the associated UTXOs spendable by anyone?
11:07 < theStack> sipa: always a pleasure to read linus rants :)
11:07 < achow101> err, not better, but rather we choose the APS one if its fee is not greater than whatever the configured max aps fee is
11:08 < achow101> pop: no, private keys are never revealed. This is just for privacy. It means that reused addresses won't be mixed with other transactions and thus reveal what utxos are belong to the same person
11:08 < pop> I see, since a single doxxed address could slowly leak utxos into transactions over time if you don't spend them all
11:08 -!- Frank0 [~Frank0@186.84.21.96] has quit [Quit: Connection closed]
11:08 < sipa> theStack: Note that C++ has a concept of "pointer to array of specified size", which allows you to actually pass arrays with size information.
11:16 -!- b_1o1 [~b_1o1@189.236.26.138] has quit [Ping timeout: 240 seconds]
11:17 < pop> Thanks everyone, achow101, Murch, glozow
11:17 -!- pop [~pop@c-71-231-107-61.hsd1.wa.comcast.net] has quit [Quit: Leaving]
11:29 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
11:35 -!- randomsats [~randomsat@cpe1033bfa9e0e7-cm1033bfa9e0e5.cpe.net.cable.rogers.com] has quit [Quit: Connection closed]
11:58 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
12:33 -!- Talkless [~Talkless@mail.dargis.net] has quit [Quit: Konversation terminated!]
12:33 < Kaizen_Kintsugi_> Thanks everyone
13:00 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Remote host closed the connection]
13:00 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
13:05 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Ping timeout: 240 seconds]
13:13 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
13:17 -!- ccdle12 [~ccdle12@243.222.90.149.rev.vodafone.pt] has quit [Quit: Client closed]
14:20 -!- drdprtrbrts [~drdprtrbr@154.0.128.44] has quit [Ping timeout: 246 seconds]
14:38 -!- siv2r[m] [~siv2rmatr@2001:470:69fc:105::fed3] has joined #bitcoin-core-pr-reviews
15:28 -!- Zenton [~user@user/zenton] has quit [Ping timeout: 256 seconds]
15:55 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Remote host closed the connection]
15:59 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@ip72-194-104-106.oc.oc.cox.net] has joined #bitcoin-core-pr-reviews
16:04 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@ip72-194-104-106.oc.oc.cox.net] has quit [Remote host closed the connection]
16:11 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
16:17 -!- svav [~svav@82-69-86-143.dsl.in-addr.zen.co.uk] has quit [Quit: Connection closed]
16:39 -!- cryptoquick0 [~cryptoqui@8.46.89.33] has joined #bitcoin-core-pr-reviews
16:40 -!- cryptoquick [~cryptoqui@8.46.89.33] has quit [Ping timeout: 240 seconds]
16:40 -!- cryptoquick0 is now known as cryptoquick
16:41 -!- effexzi [uid474242@id-474242.ilkley.irccloud.com] has quit [Quit: Connection closed for inactivity]
16:41 -!- a1ph4byte [~a1ph4byte@2600:1700:22f0:a7c0:bc46:d63f:2e5f:7485] has quit [Quit: Client closed]
16:45 -!- cryptoquick [~cryptoqui@8.46.89.33] has quit [Quit: Ping timeout (120 seconds)]
16:46 -!- cryptoquick [~cryptoqui@8.46.89.33] has joined #bitcoin-core-pr-reviews
16:47 -!- sipa [~sipa@user/sipa] has quit [Ping timeout: 240 seconds]
16:47 -!- Murch [~murch@user/murch] has quit [Ping timeout: 240 seconds]
16:47 -!- whitehorse[m] [~whitehors@2001:470:69fc:105::1:f3cf] has quit [Ping timeout: 240 seconds]
16:48 -!- whitehorse[m] [~whitehors@2001:470:69fc:105::1:f3cf] has joined #bitcoin-core-pr-reviews
16:50 -!- sipa [~sipa@user/sipa] has joined #bitcoin-core-pr-reviews
16:50 -!- Murch [~murch@user/murch] has joined #bitcoin-core-pr-reviews
17:06 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Remote host closed the connection]
17:23 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
17:27 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Ping timeout: 250 seconds]
17:41 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
18:43 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Ping timeout: 260 seconds]
18:59 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
19:13 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
19:18 -!- a1ph4byte [~a1ph4byte@2600:1700:22f0:a7c0:bc46:d63f:2e5f:7485] has joined #bitcoin-core-pr-reviews
19:20 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Remote host closed the connection]
19:21 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
19:21 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
19:56 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
20:17 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has quit [Ping timeout: 260 seconds]
20:27 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
20:31 -!- brunoerg [~brunoerg@2804:14d:5281:8ae2:899d:3cb8:f5ed:1581] has joined #bitcoin-core-pr-reviews
20:36 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
20:44 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
20:57 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Remote host closed the connection]
21:10 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
21:19 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Ping timeout: 250 seconds]
21:38 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
22:19 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has joined #bitcoin-core-pr-reviews
22:23 -!- Kaizen_Kintsugi_ [~Kaizen_Ki@2600:8802:3806:c200:5186:7734:3ca6:271c] has quit [Ping timeout: 250 seconds]
22:25 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Ping timeout: 240 seconds]
23:24 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
23:30 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has quit [Remote host closed the connection]
23:32 -!- evanlinjin [~evanlinji@gateway/tor-sasl/evanlinjin] has joined #bitcoin-core-pr-reviews
23:46 -!- belcher [~belcher@user/belcher] has quit [Ping timeout: 250 seconds]
--- Log closed Thu Apr 28 00:00:08 2022