--- Day changed Wed Sep 30 2020 00:14 -!- TheRec [~toto@drupal.org/user/146860/view] has quit [Read error: Connection reset by peer] 00:15 -!- TheRec [~toto@84-75-225-47.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews 00:15 -!- TheRec [~toto@84-75-225-47.dclient.hispeed.ch] has quit [Changing host] 00:15 -!- TheRec [~toto@drupal.org/user/146860/view] has joined #bitcoin-core-pr-reviews 01:05 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews 01:07 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 240 seconds] 01:12 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 01:48 -!- molz_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews 01:52 -!- mol_ [~mol@unaffiliated/molly] has quit [Ping timeout: 256 seconds] 02:01 -!- mol [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews 02:04 -!- molz_ [~mol@unaffiliated/molly] has quit [Ping timeout: 272 seconds] 02:18 -!- shesek [~shesek@unaffiliated/shesek] has quit [Remote host closed the connection] 02:22 -!- johnzwen_ [~johnzweng@212-186-128-119.static.upcbusiness.at] has quit [Remote host closed the connection] 02:34 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Remote host closed the connection] 02:34 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-pr-reviews 02:53 -!- johnzweng [~johnzweng@212-186-128-119.static.upcbusiness.at] has joined #bitcoin-core-pr-reviews 02:57 -!- seven_ [~seven@cpe-90-157-197-248.static.amis.net] has joined #bitcoin-core-pr-reviews 02:58 -!- johnzweng [~johnzweng@212-186-128-119.static.upcbusiness.at] has quit [Ping timeout: 272 seconds] 03:00 -!- seven__ [~seven@BNG-212-27-124-dsl.simobil.net] has quit [Ping timeout: 240 seconds] 03:10 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds] 03:12 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews 03:16 -!- musdom [~Thunderbi@113.210.188.46] has joined #bitcoin-core-pr-reviews 03:18 -!- Dora26Spencer [~Dora26Spe@static.57.1.216.95.clients.your-server.de] has joined #bitcoin-core-pr-reviews 03:18 -!- johnzweng [~johnzweng@212-186-128-119.static.upcbusiness.at] has joined #bitcoin-core-pr-reviews 03:23 -!- Dora26Spencer [~Dora26Spe@static.57.1.216.95.clients.your-server.de] has quit [Ping timeout: 256 seconds] 03:30 -!- S3RK [~s3rk@47.246.66.116] has quit [Remote host closed the connection] 03:35 -!- wumpus [~ircclient@pdpc/supporter/professional/wumpus] has joined #bitcoin-core-pr-reviews 03:38 -!- midnight [~midnight@unaffiliated/midnightmagic] has quit [Ping timeout: 272 seconds] 03:43 -!- midnight [~midnight@unaffiliated/midnightmagic] has joined #bitcoin-core-pr-reviews 03:44 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 04:17 -!- S3RK [~s3rk@47.246.66.116] has quit [Remote host closed the connection] 04:18 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 04:22 -!- S3RK [~s3rk@47.246.66.116] has quit [Ping timeout: 256 seconds] 04:31 -!- johnzweng [~johnzweng@212-186-128-119.static.upcbusiness.at] has quit [] 05:07 -!- troygiorshev [~troygiors@d67-193-140-136.home3.cgocable.net] has joined #bitcoin-core-pr-reviews 05:52 -!- musdom [~Thunderbi@113.210.188.46] has quit [Ping timeout: 256 seconds] 05:53 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has quit [Quit: jonatack] 05:57 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has joined #bitcoin-core-pr-reviews 06:41 -!- musdom [~Thunderbi@202.185.34.14] has joined #bitcoin-core-pr-reviews 07:01 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 07:05 -!- S3RK [~s3rk@47.246.66.116] has quit [Ping timeout: 240 seconds] 07:26 -!- stacie [~stacie@c-24-60-139-217.hsd1.ma.comcast.net] has joined #bitcoin-core-pr-reviews 07:48 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-pr-reviews 07:51 -!- stacie [~stacie@c-24-60-139-217.hsd1.ma.comcast.net] has quit [Ping timeout: 246 seconds] 07:51 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 256 seconds] 08:01 -!- danielabrozzoni5 [5892f803@gateway/web/cgi-irc/kiwiirc.com/ip.88.146.248.3] has joined #bitcoin-core-pr-reviews 08:02 -!- danielabrozzoni5 [5892f803@gateway/web/cgi-irc/kiwiirc.com/ip.88.146.248.3] has quit [Client Quit] 08:02 -!- danielabrozzoni [5892f803@gateway/web/cgi-irc/kiwiirc.com/ip.88.146.248.3] has joined #bitcoin-core-pr-reviews 08:06 -!- Talkless [~Talkless@mail.dargis.net] has joined #bitcoin-core-pr-reviews 08:11 < emzy> I get an error. 08:11 < emzy> $ bitcoin-cli getblock 0000000000000000000266073e5fef7d4071192fd69d340cd4f23073b0f86f61 2 08:11 < emzy> error code: -1 08:11 < emzy> error message: 08:11 < emzy> core_write.cpp:251 (TxToUniv) 08:11 < emzy> Internal bug detected: 'MoneyRange(fee)' 08:31 < emzy> older blocks are working like this: "bitcoin-cli getblock "$(bitcoin-cli getblockhash 10000)" 2" 08:31 -!- TechMiX [~TechMiX@ti0169q161-1541.bb.online.no] has joined #bitcoin-core-pr-reviews 08:36 < emzy> Block 650026 also works. Only coinbase transaction, so not undo data. 08:42 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews 09:03 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has quit [Quit: jaybny] 09:17 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews 09:18 -!- gloriazhao [uid453516@gateway/web/irccloud.com/x-ahfxotnwobjuzpgk] has joined #bitcoin-core-pr-reviews 09:21 -!- danielabrozzoni [5892f803@gateway/web/cgi-irc/kiwiirc.com/ip.88.146.248.3] has quit [Ping timeout: 240 seconds] 09:26 -!- lightlike [~lightlike@p200300c7ef1452009dfbceb31cad8988.dip0.t-ipconnect.de] has joined #bitcoin-core-pr-reviews 09:27 < wumpus> ohh good find 09:29 < emzy> wumpus: can you reproduce? I'm not sure if I doing someting wrong. 09:38 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has quit [Quit: jaybny] 09:38 -!- stacie-browser-t [183c8bd9@c-24-60-139-217.hsd1.ma.comcast.net] has joined #bitcoin-core-pr-reviews 09:40 -!- stacie [~stacie@c-24-60-139-217.hsd1.ma.comcast.net] has joined #bitcoin-core-pr-reviews 09:41 -!- elle [~ellemouto@155.93.252.70] has joined #bitcoin-core-pr-reviews 09:50 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 09:52 -!- stacie-browser-t [183c8bd9@c-24-60-139-217.hsd1.ma.comcast.net] has quit [Remote host closed the connection] 09:54 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews 09:55 -!- S3RK [~s3rk@47.246.66.116] has quit [Ping timeout: 256 seconds] 09:58 -!- test [9b5dfc46@155.93.252.70] has joined #bitcoin-core-pr-reviews 09:59 -!- test is now known as Guest30011 09:59 -!- Guest30011 [9b5dfc46@155.93.252.70] has quit [Remote host closed the connection] 09:59 -!- sunon_ [~sunon@105-213-12-21.access.mtnbusiness.co.za] has joined #bitcoin-core-pr-reviews 09:59 < robot-dreams> Starting in 10 seconds... :) 09:59 -!- epson121 [d5953e14@cm-2019.cable.globalnet.hr] has joined #bitcoin-core-pr-reviews 10:00 <@jnewbery> hi! 10:00 < robot-dreams> #startmeeting 10:00 < emzy> hi 10:00 < elle> hi :) 10:00 < robot-dreams> Hi everyone, welcome! 10:00 < amiti> hi! 10:00 < stacie> hi! first timer here 10:00 < gloriazhao> hi! 10:00 < jrawsthorne> Hi 10:00 < lightlike> hi 10:00 < dhruvm> hi! 10:00 < robot-dreams> Welcome Stacie! 10:00 < gloriazhao> ooo hello stacie so excited to see you here!!!! 10:01 < stacie> Thanks everyone! :) 10:01 -!- sunon_ [~sunon@105-213-12-21.access.mtnbusiness.co.za] has quit [Remote host closed the connection] 10:01 < robot-dreams> Friendly reminder that all questions are welcome! Feel free to jump in at any point 10:01 < felixweis> hi 10:01 < troygiorshev> hi! 10:01 < robot-dreams> Anyone else here for the first time? 10:01 < elle> yeah, my first time :) 10:01 -!- sunon [~sunon@105-213-12-21.access.mtnbusiness.co.za] has joined #bitcoin-core-pr-reviews 10:01 < robot-dreams> Welcome Elle :) 10:01 < jrawsthorne> Second time, first time not being late 10:01 < nehan> hi 10:02 < robot-dreams> Congrats jrawsthorne and great to have you back! 10:02 < robot-dreams> And last quick poll before we get into the questions, who's had a chance to review the PR? y/n 10:02 < sunon> y 10:02 < elle> y 10:02 < troygiorshev> y 10:02 < felixweis> y 10:02 < stacie> y 10:02 < jrawsthorne> y 10:03 <@jnewbery> y 10:03 < epson121> y 10:03 < nehan> yish 10:03 < amiti> n! I've taken a look but haven't properly reviewed 10:03 < emzy> y 10:03 < dhruvm> 0.5 10:04 < robot-dreams> Awesome!! Hopefully it was a relatively short / readable change, but I think there are some interesting underlying ideas. 10:04 < robot-dreams> All good Amiti, thanks for taking a look! 10:04 -!- kiwi_26 [2ec100e0@gateway/web/cgi-irc/kiwiirc.com/ip.46.193.0.224] has joined #bitcoin-core-pr-reviews 10:04 < robot-dreams> All right, let's jump in. 10:04 < robot-dreams> First off, what is `CTxUndo`? 10:05 < sipa> hi 10:05 < jrawsthorne> utxos spent by a transaction 10:05 < sunon> Class containing undo information for a CTransaction and its txins 10:05 < dhruvm> It's a convenience data structure used to unlink a block during IBD 10:06 < jonatack> hi 10:06 < robot-dreams> Excellent! I would agree with all of that 10:06 -!- kristie14 [87b432d8@135-180-50-216.fiber.dynamic.sonic.net] has joined #bitcoin-core-pr-reviews 10:06 < robot-dreams> How are using `CTxUndo` for fee calculation in the PR? 10:06 < felixweis> its a representation of the rev* data in your blocks/ directory. it allows to restore an UTXO in the set if we need to "rewind" the blockchain because a longer branch was found. remember, all spends from the shorter block chain removed the info (amount, bitcoin script) 10:08 < dhruvm> If CTxUndo is available at the call site for TxToUniv, we pass it in and use it to compute sum(outputs)-sum(inputs) 10:08 < felixweis> CTxUndo got introduced with UltraPrune invented by sipa and shipped in Bitcoin-Qt 0.8 10:08 < robot-dreams> dhruvm exactly! And question to anyone, which part of this computation involves the undo data? 10:09 < elle> the sum of the value of the inputs 10:09 -!- Paul_R [46345bd5@bras-base-mtrlpq427bw-grc-07-70-52-91-213.dsl.bell.ca] has joined #bitcoin-core-pr-reviews 10:09 < robot-dreams> elle: yes, exactly 10:09 < Paul_R> hi, excuse my tardiness. 10:09 < robot-dreams> felixweis thanks for the extra context! 10:09 < Paul_R> is there anyway I can read the backlog of this session? 10:10 < robot-dreams> hi Paul_R, welcome! 10:10 < Paul_R> thx robot-dreams 10:10 < robot-dreams> It'll be posted on the website after the session, so you'll be able to see the full log 10:10 < Paul_R> ok 10:10 < robot-dreams> felixweis: btw thanks for writing the commit that added this! 10:11 < felixweis> thanks *you* for picking it back up! 10:11 < robot-dreams> Quick question to make sure we're all on the same page, why doesn't a coinbase transaction have Undo data? 10:12 < sunon> There are no txins right? 10:12 < dhruvm> Because there's no UTXO 10:12 < sipa> no effective txins 10:12 < elle> It doesnt have an input that needs to be undone 10:12 < robot-dreams> sunon dhruvm sipa elle exactly! sounds like we're all on the same page here 10:12 < sipa> there is a CTxIn in a coinbase, but it's not actually spending any preexisting UTXO 10:12 < sunon> Ah yeah so not a relevant txin 10:13 < robot-dreams> An interesting follow-up question (outside the scope of this session) is: what are all the uses of the coinbase CTxIn? 10:13 < robot-dreams> And finally, where else is CTxUndo used? dhruvm already mentioned unlinking a block during IBD; is there anything else? 10:14 < sunon> Extra nonce space? 10:14 < sipa> and BIP34 10:15 < sunon> I found it being used in UpdateCoins() in validation but have no idea why to be honest. 10:15 < sipa> that's just where it's constructed 10:16 < jrawsthorne> For the filter index 10:16 < sunon> ah I see 10:17 < robot-dreams> sunon: Yup, I agree there's not a lot of context in UpdateCoins itself, but I found looking at where UpdateCoins itself got called to be helpful 10:17 < felixweis> BIP34 encodes the block height in the coinbase scriptSig field. so coinbase txids are still unique. previously they sometimes did override old coins. 10:18 < jrawsthorne> For previous question, although not related to consensus pools use it to mark their blocks 10:18 -!- Myrtle63Kuhlman [~Myrtle63K@static.57.1.216.95.clients.your-server.de] has joined #bitcoin-core-pr-reviews 10:19 < robot-dreams> jrawsthorne Nice, that seems like another interesting use of the undo data! Something else we can follow up on if we have time later, "what exactly is the filter index and how does that work" 10:19 < robot-dreams> But let's proceed for now. 10:19 < robot-dreams> This might be my favorite question for today's session: 10:19 < robot-dreams> What's the IO cost to read Undo data for a block? 10:19 < Paul_R> by override, could you equally say the over-wrote the old coins? ie. they ceased to exist anymore? seems like i would have heard of that bug by now, so i'm expecting override to not mean overwrite 10:20 < Paul_R> @feli 10:20 < Paul_R> felixweis 10:21 < sipa> Paul_R: read BIP30 10:21 < dhruvm> `UndoReadFromDisk` would imply it's not expected to be in memory and is being read from disk when/if the block has not been pruned. 10:21 < Paul_R> @sipa ok 10:21 < dhruvm> But I haven't actually looked inside that function yet 10:22 < robot-dreams> dhruvm: Yeah, I agree with that! We are gonna be hitting disk. 10:22 < felixweis> I believe its fairly low thanks to the excellent design of UtraPrune™. we only need to read a single additional file which has around ~100 kB of data per block to get all the amount information of all the txins spend in that particular block. 10:23 < dhruvm> And this is only in response to a privileged rpc call, so there's probably no abuse concerns. 10:23 -!- Myrtle63Kuhlman [~Myrtle63K@static.57.1.216.95.clients.your-server.de] has quit [Ping timeout: 258 seconds] 10:24 < robot-dreams> felixweis exactly! 10:24 -!- sunon_ [~sunon@105-213-12-21.access.mtnbusiness.co.za] has joined #bitcoin-core-pr-reviews 10:24 < dhruvm> We don't need to worry about cascading issues even if the read is slow. 10:24 < robot-dreams> dhruvm great point, we probably don't want a P2P message that says "hey peer, read from disk" 10:24 -!- sunon_ [~sunon@105-213-12-21.access.mtnbusiness.co.za] has quit [Remote host closed the connection] 10:24 < sipa> i mean: that exists 10:24 < Paul_R> where can i read more about priveleged rpc calls (ie. examples of such) 10:24 < sipa> it's called getdata 10:25 < robot-dreams> sipa: I realized that shortly after I said my previous statement 10:25 < jrawsthorne> I can get your node to give me the entire blockchain over and over again right 10:25 < robot-dreams> Yes jrawsthorne! I stand corrected 10:25 < sipa> Paul_R: he means RPC is only ever exposed to privileged software, so we don't need to worry about DoS potential, i assume 10:26 < dhruvm> sipa: yes, that's what i meant 10:26 < robot-dreams> We'll edit that in post :) 10:26 < sipa> there is no distinction between privileged and unprivileged RPC - it's just that RPC is only intended to be exposed to trusted software 10:26 < sipa> (it can send a shutdown RPC too...) 10:26 < Paul_R> ah 10:27 < robot-dreams> One follow-up question here is, "what are the DoS mitigation mechanisms getdata has that this RPC doesn't" 10:27 < dhruvm> we mitigate the DoS potential of getdata disk access using peer reputation though, i imagine? 10:27 < sipa> dhruvm: nope, the only protection that currently exists is just that an attacker would need to waste N bytes bandwidth to make us read N bytes from disk 10:28 < dhruvm> huh, interesting! 10:28 < robot-dreams> So it's a matter of symmetry between cost to the attacker and cost to the node? 10:28 < sipa> (which is one of the reasons why BIP37 was disabled by default - it enabled an attacker to ask for a block to be read from disk without pretty much any bandwidth at all) 10:29 < robot-dreams> felixweis: going back to IO cost, yup! We read a contiguous section of a `rev?????.dat` file, which contains the serialized `CTxUndo` data (a vector of UTXO info) 10:30 < robot-dreams> I got an estimate of 64kb by adding up the size of all the rev* files and dividing by the number of blocks. Same order of magnitude as your 100kb estimate. 10:30 <@jnewbery> there is some (limited) protection from -maxuploadtarget. It's more to protect bandwidth, but I think probably also protects disk reads indirectly 10:31 -!- sunon [~sunon@105-213-12-21.access.mtnbusiness.co.za] has quit [] 10:32 < jrawsthorne> Didn't know about that flag 10:32 < robot-dreams> dhruvm: great point to consider DoS potential and sipa: jnewbery: thanks for the extra context on this! 10:33 < robot-dreams> Any final thoughts on DoS before we talk about a world without Undo data? 10:33 < lightlike> sipa: don't we only process a single GETDATA block request each cycle in ProcessMessages - this should mitigate DOS concerns a bit. 10:33 < robot-dreams> Thanks lightlike! 10:34 < emzy> How to test this PR manually. Like with bitcoin-cli? I tested "getblock" and will get an error. 10:34 < sipa> lightlike: somewhat, indeed 10:34 < robot-dreams> OK moving onto the next question, how could we calculate the fee if Undo data didn’t exist for a block, and what’s the IO cost? 10:35 < jrawsthorne> If we had txindex we could find each input by txid and output index 10:35 < emzy> ' bitcoin-cli getblock "$(bitcoin-cli getblockhash 650683)" 2 ' I will get a 'core_write.cpp:251 (TxToUniv) Internal bug detected: 'MoneyRange(fee)'' 10:36 < robot-dreams> emzy: Great catch, thanks for finding that! I definitely want to look into that afterwards. 10:36 < robot-dreams> jrawsthorne: Great! How does that cost compare to reading `rev?????.dat`? 10:37 < robot-dreams> And jrawsthorne described the case where we already have a txindex. How could we get the fee if we didn't turn on txindex? 10:39 <@jnewbery> without an undo file, you'd need to somehow find the txouts in the block files, and they'd be scattered across many blocks 10:39 < jrawsthorne> A lot more expensive, with undo we can read it all at once but with this we'd have to get each tx individually and then find the correct output 10:39 < jrawsthorne> Is there a function for just getting a specific output or is it just the whole tx? 10:39 < robot-dreams> jnewbery: jrawsthorne: right! Even with a txindex that doesn't sound too fun 10:39 < sipa> jrawsthorne: not even 10:40 < sipa> you'd need to read through the entire blockchain 10:40 < nehan> i think you'd have to scan the whole blockchain 10:40 < dhruvm> So `TxToUniv` seems to have access to `CTransaction`. `CTransaction` has `vins` and `vouts`. `vouts` have amount, but `cTxIn` does not - it points to prevout. 10:40 < felixweis> i dont think you can find the the prev_out txs without scanning trough every block 10:40 < robot-dreams> With a txindex: for each spent UTXO we'd need one read from LevelDB, then one read from disk 10:40 < sipa> indeed, that's exactly what the txindex is for 10:40 < robot-dreams> sipa: felixweis: Yeah, I'd agree that without a txindex you'd just have to scan 10:40 < nehan> you could do one scan looking for all the outputs 10:40 < sipa> without it, you need to reconstruct it from scratch 10:41 < sipa> nehan: basically building the txindex :) 10:41 < robot-dreams> All right, so it sounds like using Undo data is a better strategy than "scan the entire blockchain" :) 10:41 < dhruvm> little bit 10:41 < dhruvm> :) 10:41 < felixweis> nehan: yes building a list first of all inputs and then scaning through the chain is a lot mor efficient than the other way around :-) 10:41 < robot-dreams> OK, going onto some more open-ended and possibly stylistic questions here. 10:42 < robot-dreams> What does CHECK_NONFATAL do, and how does it compare to other options we might have like asserts? 10:42 < nehan> question: why not put amounts and/or scriptpks in inputs, besides space? (or is it just space) 10:42 < robot-dreams> When should we consider using each option? 10:42 < jrawsthorne> throws a specific exception that is caught by the rpc code and sent as an error in the response 10:43 < jrawsthorne> if a condition is not met 10:43 < nehan> given that all nodes store undo data anyway, why not just... put it in the blocks? 10:43 < sipa> nehan: define "put it in the blocks"? 10:43 < dhruvm> nehan: when the same data is stored in two places, could it increase the possibility of bugs where the two are inconsistent? 10:43 < sipa> on disk? in p2p? in commitment structure? 10:44 < nehan> sipa: good question! hmm. i guess in this case, in the blocks 10:44 < nehan> that would help with reorgs 10:44 < sipa> nehan: you haven't answered my question :p 10:44 < robot-dreams> nehan: Interesting design question! Yeah, "if we were redesigning from scratch would we just put rev????? data right next to block data" 10:44 < nehan> oh, i meant on disk 10:45 < felixweis> you mean the blk????? files 10:45 < sipa> nehan: i'd say arguably we do 10:45 < sipa> it's just split between blk and rev files 10:45 < sipa> so that the rev files can be deleted sooner 10:45 < nehan> what about in p2p? 10:45 < nehan> i guess the answer to that is bw 10:45 < stacie> when talking about txindex, is it defined as `block_height:index_of_tx_in_block` or `txid:index_of_output` (in terms of finding more details on a vin). I'm guessing it's the former? 10:46 < robot-dreams> jrawsthorne: Yup! Returning an error instead of crashing the node. 10:46 < sipa> nehan: well, that's what utreexo (and cory's predecessor idea to it) do ;) 10:46 < nehan> sipa: yes, was just thinking of that 10:46 < Paul_R> @nehan 'bw'? 10:46 < sipa> bandwidth 10:47 < nehan> here is cory's UHS idea for those who haven't seen it: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-May/015967.html 10:47 < robot-dreams> OK, next is a "UX" question: Knowing the transaction fee is useful, but does only returning it in some circumstances actually provide a better client experience, or is it preferable for clients to always get exactly what they expect? 10:48 < felixweis> stacie: index is I believe txid -> block_height 10:49 < robot-dreams> stacie: Yeah, great question! I agree with felixweis, I think specifically the txindex tells you the file where you can get the block data, as well as some relevant offsets into the file to find the data 10:49 < robot-dreams> Source: https://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair/29418#29418 10:50 <@jnewbery> the txindex interface takes a txid and returns a blockhash and a CTransactionRef object: https://github.com/bitcoin/bitcoin/blob/72affcb16cad45bd9e08ca163b2147fd01b84b70/src/index/txindex.h#L42-L48 10:50 < lightlike> another question: while this PR shows the fees for each tx: would it maybe be useful to also display the total amount of fees spent in a block in the getblock RPC? 10:50 < sipa> it's txid -> (block_file_num, offset_of_block_in_file, offset_of_tx_from_block) 10:50 <@jnewbery> internally, the index actually stores txid->diskpos (ie where in a block file the tx can be found) 10:50 < sipa> iirc 10:51 < robot-dreams> sipa: your recollection agrees with what you previously wrote in the linked StackExchange answer above! 10:51 < felixweis> so pretty much the location of the first byte of the transaction on disk? 10:51 < elle> lightlike: i think that that can already be implied from the coinbase transaction? coinbaseAmt minus blockreward 10:52 < nehan> sipa: i don't know what you mean by the third option, "commitment structure". input data is already committed to in a transaction. somewhere else? 10:52 <@jnewbery> oops actually the name of the object is CDiskTxPos 10:52 < robot-dreams> lightlike: elle: great points! I don't have enough context to know whether we want to provide the extra convenience of such a calculation, or assume RPC clients can do that on their own 10:53 < sipa> nehan: this may be a bit of a tangent, but i think it's easiest to think of blocks and transactions as abstract data structures that are defined by their commitment structure (so a transaction is defined by its txid, witness transaction by its wtxid, block by its block hash, ...), and how/what is exactly stored/sent is implementation detail 10:54 < lightlike> elle: right, that makes more sense :-) 10:54 < sipa> nehan: so in that sense, sennding input data along with blocks isn't making it part of "the block" - it's just auxiliary data that happens to be sent/stored along with it 10:55 < robot-dreams> All right, we have 5 minutes left. 10:55 < dhruvm> robot-dreams: in response to the UX question. when representing the same data in two ways(in this case as fee, and sum(op) - sum(ip)), i worry what happens if a bug makes the two inconsistent. which representation is truth for the client? should a low-level protocol only represent the bare minimum and leave the rest to clients? should the protocol minimize computation needed from clients? 10:55 < dhruvm> tough question. 10:55 < robot-dreams> At this point does anyone have any general questions? 10:55 <@jnewbery> 5 minutes left. Don't be shy! 10:56 < sipa> dhruvm: what do you mean by representing in that context? the fee is never stored explicitly afaik 10:57 < jonatack> if anyone's interested, there was a related review club session in April, "Flush undo files after last block write (validation)" https://bitcoincore.reviews/17994 10:57 < dhruvm> sipa: I mean from the client's perspective. If the fee and sum(op)-sum(ip) ever disagree, the clients should prefer the latter. But explicitly sending them the fee makes it possible that they might not? 10:57 < felixweis> I'm curious about how people feel about Question 6. Should we just return the fee always and not introduce another "verbosity" level? even if there are speed penatlys for the rpc? 10:58 < robot-dreams> felixweis: Yeah, very interesting question. On a similar note I'm also curious if people think we should calculate the fee when a txindex (but not block undo data) is available. 10:59 <@jnewbery> robot-dreams, I don't think that's a realistic scenario. The txindex requires the entire blockchain (i.e. no pruning) 10:59 < felixweis> It would of course not be possible if you have your node pruned beyond the requested block but then wold you request getblock anyway? 10:59 < sipa> dhruvm: what is "the fee" 10:59 < sipa> dhruvm: the fee is defined as sum(out)-sum(in), so i don't know how they could be different 10:59 <@jnewbery> so if you have txindex, you should also have all the undo files. We only prune undo files when we prune the equivalent block files 10:59 < robot-dreams> jnewbery: felixweis: fair points! 11:00 < robot-dreams> All right, we're out of time for today's session. 11:00 < robot-dreams> Thanks so much everyone for joining! 11:00 < dhruvm> sipa: yeah i guess it's just my ocd when representing the same data in two ways. I've experienced bugs from that in the past. Perhaps not something to worry about. 11:00 < robot-dreams> Especially those of you who were able to join us for the first time, welcome again, and I hope to see you around at future sessions! 11:00 <@jnewbery> Thanks robot-dreams! 11:00 < felixweis> thanks robot-dreams! and everynone else participating and lurking. 11:00 < nehan> thanks robot-dreams! 11:00 < sipa> dhruvm: but i still don't understand what you mean by representing 11:01 < stacie> thanks robot-dreams:! Learned a lot 11:01 < robot-dreams> #endmeeting 11:01 < sipa> the fee isn't stored 11:01 < emzy> thanks robot-dreams 11:01 < troygiorshev> thanks robot-dreams! 11:01 < lightlike> thanks! 11:01 < sipa> it's computed as sum(out)-sum(in) whenever needed 11:01 < kristie14> thanks! this was great as a lurker 11:01 < Paul_R> thanks robot-dreams 11:01 < elle> thanks robot-dreams! 11:01 < felixweis> the fee is implicit 11:01 < sipa> thanks! 11:01 < dhruvm> sipa: right. if there's a future bug in that logic is what I'm concerned about. 11:01 < jonatack> thanks robot-dreams, great session! 11:01 -!- Paul_R [46345bd5@bras-base-mtrlpq427bw-grc-07-70-52-91-213.dsl.bell.ca] has quit [Remote host closed the connection] 11:01 < sipa> dhruvm: in what logic? 11:01 < elle> \win 1 11:02 < robot-dreams> dhruvm: Am I understanding that you're talking about "fee for the entire block vs. fee for each individual transaction"? 11:02 -!- kristie14 [87b432d8@135-180-50-216.fiber.dynamic.sonic.net] has quit [Remote host closed the connection] 11:02 < dhruvm> robot-dreams: no, not that. 11:04 < robot-dreams> Got it, thanks 11:05 < emzy> robot-dreams: You have time for my problem? 11:05 < dhruvm> sipa: I am thinking of a future scenario where the server computes fee=sum(op)-sum(ip) incorrectly due to a bug. Now the client receives fee that is different than the formula. 11:05 < robot-dreams> emzy: I will in a bit! 11:05 < felixweis> emzy: i got the same problem 11:06 < emzy> felixweis: good to know :) 11:06 < dhruvm> sipa: But I can also see how that's just paranoia. If there's a bug, we'll fix it. It's not going to be at the storage level since like you said we are not storing anything differently. 11:08 < sipa> dhruvm: who is the server and the client? fees are never sent explicitly 11:08 < dhruvm> sipa: rpc server, rpc client 11:09 < sipa> oh 11:09 < sipa> the rpc client never computes anything 11:09 < sipa> so there can be discrepancy 11:09 < sipa> *no 11:10 < dhruvm> i see. because it merely prints it out. 11:10 < jonatack> dhruvm: the only client-side computation occurs in src/bitcoin-cli.cpp: -getinfo, -netinfo, -generate 11:11 < dhruvm> what about when the rpc client is third party software? 11:11 < sipa> dhruvm: if they implement fee calculation incorrectly, then they have a bug and should fix it 11:11 < sipa> i don't see how that's any worse (or better) than doing whatever else they're doing incorrectly 11:11 < dhruvm> right, that's if we let them do the computation. in this case we are choosing to do the computation for them. 11:12 < sipa> dhruvm: i think you're focussing too much on a consistency problem here; the goal isn't consistency - it's correctness 11:12 < sipa> ask yourself: would it be better or worse if somehow both the server and the client simultaneously introduce the same "bug" 11:12 < sipa> if it's better, then it's a consistency problem 11:12 < sipa> if it's worse, then it's a correctness proble 11:13 < sipa> the goal isn't that two computations end up with the same result per se - the goal is that the fee is correctly calculated, everywhere, and that implies that all implementations will come to the same conclusion 11:13 < sipa> but just ending up with the same conclusion isn't the actual goal 11:14 < dhruvm> oh wow, that's a mindset shift. that makes a TON of sense. 11:15 < sipa> something like bitcoin block validation is in many ways not a correctness problem - there are plenty of weird deviations from the consensus rules that could be accidentally made, as long as everyone makes the same deviations 11:15 < sipa> but here "the fee" is a well defined thing, and if you implement something that doesn't match the definition, you're wrong, period 11:15 < dhruvm> because there's a mutual understanding of the protocol, and that's what matters more than any implementation detail. 11:15 < emzy> sipa: but someone needs to be the reference. 11:16 < sipa> emzy: i hereby decree that The Fee is defined as the sum of transaction outputs' values minus the sum of the values of the outputs spent by the transaction's input. 11:16 < sipa> Done. 11:16 < emzy> :) 11:16 < dhruvm> emzy: i think what sipa is saying is that it's well understood sum(op)-sum(ip) is the definition of fee. now, if there's a bug on rpc server, or rpc client, that doesn't matter as much. 11:16 < emzy> Yes this is a easy one. 11:17 < dhruvm> Thank you for teaching us, sipa! 11:17 < sipa> np 11:20 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has quit [Quit: jaybny] 11:21 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews 11:27 -!- elle [~ellemouto@155.93.252.70] has quit [Quit: leaving] 11:31 -!- troygiorshev [~troygiors@d67-193-140-136.home3.cgocable.net] has quit [Ping timeout: 272 seconds] 11:56 -!- lightlike [~lightlike@p200300c7ef1452009dfbceb31cad8988.dip0.t-ipconnect.de] has quit [Quit: Leaving] 11:57 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 12:01 -!- S3RK [~s3rk@47.246.66.116] has quit [Ping timeout: 240 seconds] 12:27 -!- Talkless [~Talkless@mail.dargis.net] has quit [Quit: Konversation terminated!] 12:57 -!- stacie [~stacie@c-24-60-139-217.hsd1.ma.comcast.net] has quit [Quit: stacie] 13:04 -!- epson121 [d5953e14@cm-2019.cable.globalnet.hr] has quit [Remote host closed the connection] 13:39 -!- kiwi_26 [2ec100e0@gateway/web/cgi-irc/kiwiirc.com/ip.46.193.0.224] has quit [Quit: Connection closed] 13:50 -!- sunon [~sunon@105-213-12-21.access.mtnbusiness.co.za] has joined #bitcoin-core-pr-reviews 13:50 -!- sunon [~sunon@105-213-12-21.access.mtnbusiness.co.za] has quit [Remote host closed the connection] 14:12 -!- b10c_ [~b10c@fttx-pool-217.61.159.138.bambit.de] has joined #bitcoin-core-pr-reviews 14:16 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has joined #bitcoin-core-pr-reviews 14:17 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has quit [Client Quit] 14:19 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has joined #bitcoin-core-pr-reviews 14:22 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has quit [Client Quit] 14:22 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has joined #bitcoin-core-pr-reviews 14:31 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has quit [Quit: leaving] 14:31 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has joined #bitcoin-core-pr-reviews 14:47 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has quit [Quit: leaving] 14:54 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has joined #bitcoin-core-pr-reviews 15:05 -!- b10c__ [~b10c@fttx-pool-217.61.159.138.bambit.de] has joined #bitcoin-core-pr-reviews 15:07 -!- b10c_ [~b10c@fttx-pool-217.61.159.138.bambit.de] has quit [Ping timeout: 240 seconds] 15:07 -!- b10c [~b10c@static.43.111.76.144.clients.your-server.de] has quit [Quit: Lost terminal] 15:09 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has quit [Quit: jaybny] 15:11 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds] 15:12 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews 15:25 -!- b10c [~b10c@144.76.111.43] has joined #bitcoin-core-pr-reviews 15:26 -!- b10c [~b10c@144.76.111.43] has quit [Client Quit] 15:27 -!- valwal_ [sid334773@gateway/web/irccloud.com/x-quzwnfdrpkglflcc] has joined #bitcoin-core-pr-reviews 15:28 -!- b10c [~b10c@2a01:4f8:192:612a:216:3eff:fef3:dc6a] has joined #bitcoin-core-pr-reviews 15:38 -!- b10c__ [~b10c@fttx-pool-217.61.159.138.bambit.de] has quit [Quit: Leaving] 15:40 -!- musdom [~Thunderbi@202.185.34.14] has quit [Quit: musdom] 15:42 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has quit [Ping timeout: 240 seconds] 15:59 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 16:04 -!- S3RK [~s3rk@47.246.66.116] has quit [Ping timeout: 272 seconds] 16:32 -!- seven_ [~seven@cpe-90-157-197-248.static.amis.net] has quit [Ping timeout: 240 seconds] 17:31 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews 17:45 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 18:27 -!- jaybny [~jaybny@c-73-162-160-252.hsd1.ca.comcast.net] has quit [Quit: jaybny] 18:44 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has joined #bitcoin-core-pr-reviews 20:34 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has quit [Remote host closed the connection] 20:34 -!- davterra [~davterra@gateway/tor-sasl/tralfaz] has joined #bitcoin-core-pr-reviews 21:00 -!- achow101 [~achow101@unaffiliated/achow101] has quit [Quit: Bye] 21:00 -!- achow101 [~achow101@unaffiliated/achow101] has joined #bitcoin-core-pr-reviews 22:09 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has quit [Ping timeout: 240 seconds] 22:54 -!- S3RK [~s3rk@47.246.66.116] has quit [Remote host closed the connection] 22:55 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 22:57 -!- geahaad [c0918dbd@192.145.141.189] has joined #bitcoin-core-pr-reviews 22:59 -!- geahaad [c0918dbd@192.145.141.189] has quit [Remote host closed the connection] 23:00 -!- S3RK [~s3rk@47.246.66.116] has quit [Ping timeout: 258 seconds] 23:01 -!- geahaad [uid467208@gateway/web/irccloud.com/x-itvpnshwcbdejwrh] has joined #bitcoin-core-pr-reviews 23:07 -!- geahaad is now known as geahaad_backup 23:09 -!- geahaad_backup is now known as geahaad 23:09 -!- geahaad [uid467208@gateway/web/irccloud.com/x-itvpnshwcbdejwrh] has quit [] 23:09 -!- geahaad [uid467208@gateway/web/irccloud.com/x-icgqhsrxjolxvulk] has joined #bitcoin-core-pr-reviews 23:14 -!- seven_ [~seven@cpe-90-157-197-248.static.amis.net] has joined #bitcoin-core-pr-reviews 23:16 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Ping timeout: 240 seconds] 23:25 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-core-pr-reviews 23:37 -!- S3RK [~s3rk@47.246.66.116] has joined #bitcoin-core-pr-reviews 23:43 -!- S3RK [~s3rk@47.246.66.116] has quit [Ping timeout: 272 seconds] 23:54 -!- Netsplit *.net <-> *.split quits: Aleru, fjahr, felixweis, sharknado9000[m], djinni`