--- Day changed Wed May 27 2020
00:12 -!- jonatack_ [~jon@37.170.255.11] has quit [Ping timeout: 246 seconds]     
00:13 -!- jonatack_ [~jon@184.75.223.203] has joined #bitcoin-core-pr-reviews     
00:23 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has joined #bitcoin-core-pr-reviews     
00:53 -!- jonatack_ [~jon@184.75.223.203] has quit [Quit: jonatack_]     
00:58 -!- jonatack [~jon@184.75.223.203] has joined #bitcoin-core-pr-reviews     
01:12 -!- jkczyz_ [sid419941@gateway/web/irccloud.com/x-wjxmksunuwrmguxl] has joined #bitcoin-core-pr-reviews     
01:20 -!- Netsplit *.net <-> *.split quits: jkczyz
01:20 -!- jkczyz_ is now known as jkczyz     
02:26 -!- provoostenator [~quassel@provoostenator.sprovoost.nl] has joined #bitcoin-core-pr-reviews     
03:05 -!- Guido42Mayer [~Guido42Ma@static.57.1.216.95.clients.your-server.de] has joined #bitcoin-core-pr-reviews     
03:09 -!- harrigan [~harrigan@ptr-93-89-242-235.ip.airwire.ie] has quit [Read error: Connection reset by peer]     
03:12 -!- harrigan [~harrigan@ptr-93-89-242-235.ip.airwire.ie] has joined #bitcoin-core-pr-reviews     
03:12 -!- Guido42Mayer [~Guido42Ma@static.57.1.216.95.clients.your-server.de] has quit [Ping timeout: 260 seconds]     
03:13 -!- MajkiMajk [57ce3181@87-206-49-129.dynamic.chello.pl] has joined #bitcoin-core-pr-reviews     
03:14 -!- MajkiMajk [57ce3181@87-206-49-129.dynamic.chello.pl] has quit [Remote host closed the connection]     
03:15 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has quit [Remote host closed the connection]     
03:24 -!- dfmb_ [~dfmb_@unaffiliated/dfmb/x-4009105] has joined #bitcoin-core-pr-reviews     
03:40 -!- jonatack [~jon@184.75.223.203] has quit [Quit: jonatack]     
03:46 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has joined #bitcoin-core-pr-reviews     
03:47 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has joined #bitcoin-core-pr-reviews     
04:11 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has quit [Ping timeout: 240 seconds]     
04:14 -!- jonatack [~jon@184.75.221.43] has joined #bitcoin-core-pr-reviews     
04:52 -!- pinheadmz [~pinheadmz@96.47.229.196] has joined #bitcoin-core-pr-reviews     
05:42 -!- troygiorshev [~troygiors@CPEdcef09a0ed55-CM0c473d74be00.cpe.net.cable.rogers.com] has joined #bitcoin-core-pr-reviews     
06:13 -!- wullon5 [~wullon@241.243.86.88.rdns.comcable.net] has quit [Ping timeout: 272 seconds]     
08:21 -!- adiabat [~adiabat@63.209.32.102] has joined #bitcoin-core-pr-reviews     
08:22 -!- tarboss [54a034ab@gateway/web/cgi-irc/kiwiirc.com/ip.84.160.52.171] has joined #bitcoin-core-pr-reviews     
08:41 -!- Talkless [~Talkless@hst-227-49.splius.lt] has joined #bitcoin-core-pr-reviews     
08:46 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has joined #bitcoin-core-pr-reviews     
08:49 -!- wullon5 [~wullon@182.234.86.88.rdns.comcable.net] has joined #bitcoin-core-pr-reviews     
08:51 -!- tarboss [54a034ab@gateway/web/cgi-irc/kiwiirc.com/ip.84.160.52.171] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client]     
08:52 -!- tarboss [54a034ab@gateway/web/cgi-irc/kiwiirc.com/ip.84.160.52.171] has joined #bitcoin-core-pr-reviews     
09:03 -!- wullon5 [~wullon@182.234.86.88.rdns.comcable.net] has quit [Quit: The Lounge - https://thelounge.chat]     
09:05 -!- MajkiMajk [57ce3181@87-206-49-129.dynamic.chello.pl] has joined #bitcoin-core-pr-reviews     
09:11 -!- tarboss [54a034ab@gateway/web/cgi-irc/kiwiirc.com/ip.84.160.52.171] has left #bitcoin-core-pr-reviews []     
09:12 < jnewbery> we'll get started in about 45 minutes
09:20 -!- vasild_ [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews     
09:23 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]     
09:23 -!- vasild_ is now known as vasild     
09:24 -!- troygiorshev [~troygiors@CPEdcef09a0ed55-CM0c473d74be00.cpe.net.cable.rogers.com] has quit [Ping timeout: 256 seconds]     
09:26 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has joined #bitcoin-core-pr-reviews     
09:29 -!- Talkless [~Talkless@hst-227-49.splius.lt] has quit [Ping timeout: 246 seconds]     
09:34 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has quit [Quit: The Lounge - https://thelounge.chat]     
09:35 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has joined #bitcoin-core-pr-reviews     
09:36 -!- thomasb06 [~user@eth-west-pareq2-46-193-0-224.wb.wifirst.net] has joined #bitcoin-core-pr-reviews     
09:44 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has quit [Quit: The Lounge - https://thelounge.chat]     
09:45 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has joined #bitcoin-core-pr-reviews     
09:49 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has quit [Remote host closed the connection]     
09:50 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has joined #bitcoin-core-pr-reviews     
09:51 -!- shaunsun [shaunsun@gateway/vpn/privateinternetaccess/shaunsun] has joined #bitcoin-core-pr-reviews     
09:51 -!- troygiorshev [~troygiors@CPEdcef09a0ed55-CM0c473d74be00.cpe.net.cable.rogers.com] has joined #bitcoin-core-pr-reviews     
09:55 -!- r251d [~r251d@162-196-59-192.lightspeed.irvnca.sbcglobal.net] has joined #bitcoin-core-pr-reviews     
10:00 < fjahr> #startmeeting
10:00 < fjahr> hi     
10:00 < adiabat> hi     
10:00 < raj_149> hi     
10:00 < thomasb06> hi     
10:00 < nehan> hi     
10:00 < troygiorshev> hi     
10:00 < r251d> hi     
10:00 < jnewbery> hi     
10:00 < kanzure> hi     
10:00 < sipa> ~hi     
10:00 < fjahr> Hi everyone, welcome to this weeks pr review club. I am happy to share this one with you, since I have been working on this for a while.
10:00 < willcl_ark> hi     
10:00 -!- tarboss [~tarboss@p54a034ab.dip0.t-ipconnect.de] has joined #bitcoin-core-pr-reviews     
10:00 < fjahr> Feel free to ask questions any time, there can be multiple topics discussed at the same time.
10:00 < fjahr> Who had a chance to review the PR? (y/n)
10:01 < michaelfolkson> hi     
10:01 < troygiorshev> n     
10:01 < raj_149> y     
10:01 < nehan> y     
10:01 < adiabat> y     
10:01 < sipa> y     
10:01 < willcl_ark> y     
10:01 -!- uproar [~uproar@178.162.222.42.adsl.inet-telecom.org] has joined #bitcoin-core-pr-reviews     
10:01 < raj_149> what does bogosize mean?
10:02 < sipa> it's a vague metric for size of the utxo set, but it doesn't have any actual meaning
10:02 -!- ecurrencyhodler [68ac287f@cpe-104-172-40-127.socal.res.rr.com] has joined #bitcoin-core-pr-reviews     
10:02 < sipa> it's arbitrarily chosen constants
10:02 -!- gzhao408 [49fcfb03@c-73-252-251-3.hsd1.ca.comcast.net] has joined #bitcoin-core-pr-reviews     
10:02 < fjahr> "A meaningless metric for UTXO set size" says the help :p
10:02 < jkczyz> hi     
10:03 < sipa> it's inspired by "bogomips" in the linux /proc/cpuinfo which is a meaningless indicator for processing speed
10:03 < raj_149> sipa: fjahr oh.. :p
10:03 < fjahr> but good question, definitely something to consider removing :)
10:03 < sipa> all it means is larger number -> larger utxo set, somewhat
10:03 -!- theStack [~honeybadg@vps1648322.vs.webtropia-customer.com] has joined #bitcoin-core-pr-reviews     
10:03 < theStack> hi     
10:03 < sipa> it's useful for comparison
10:03 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has quit [Quit: The Lounge - https://thelounge.chat]     
10:03 < sipa> the number just doesn't have any physical meaning
10:04 < sipa> there is also a field for the on-disk size, but that's nondeterministic
10:04 < fjahr> Let's start with the questions I came up with, they are focussed on conceptual understanding, but definitely throw in more technical questions! What does “incremental hashing” mean? Which of its properties are interesting for our use case?
10:04 -!- rish [dfbf246a@223.191.36.106] has joined #bitcoin-core-pr-reviews     
10:05 < fjahr> sipa: but is it used by anyone? I could not figure that out tbh
10:05 < michaelfolkson> Updating the hash value rather than calculating a new hash from scratch
10:06 < raj_149> fjahr: adding new items into the hash input set without redoing the full hash. Easy to add and remove items from the set seems disreable for this purpose.
10:06 < uproar> incremental hashing: change in work to recompute message digest is a function of how much the input message has changed
10:06 -!- rish [dfbf246a@223.191.36.106] has left #bitcoin-core-pr-reviews []     
10:06 < uproar> is my attempted definition
10:07 < jnewbery> a hash function that takes a set of items and returns a digest, with operations to add and remove items from the set
10:07 < uproar> and that that change is proportional to input message delta
10:07 < gzhao408> particularly useful here, since the UTXO set is very large and unordered - we frequently add and remove items in no particular order
10:07 < fjahr> raj_149: not sure if you mean to say that but it does need to be a set in the strict definition :)
10:08 < sipa> i think a better term is homomorphic set hashing
10:08 < uproar> unpacking that phrase would help
10:08 < sipa> incremental just means you can easily add items, but doesn't imply anything about removal or the order in which you can do so
10:09 < fjahr> true, that is the better phrase that includes the properties I was hinting at in the second question
10:09 < sipa> probably not worth the semantics discussion :)
10:09 < willcl_ark> that you can recompute the hash based on the old hash, after adding/removing items
10:10 < theStack> does a merkle tree also count as incremental hash structure? it's quite easy to add a new leaf item and propagate up to the root hash
10:10 < fjahr> I think these answer were all good, lets move on: What were the use cases for RPC gettxoutsetinfo described in the resource links? Can you think of others? Which use cases are the most relevant to you?
10:10 < sipa> theStack: if you treat the entire merkle tree as the "hash", i'd say yes - but it's not a compact one
10:11 < ecurrencyhodler> Can be used to validate snapshots for AssumeUTXO and also used for BTCPayServer's FastSync.
10:11 < willcl_ark> I think it might be nice for an SPV wallet, along with checking valid headers (with most work) were provided, to also query multiple nodes' UTXO set hashes
10:11 < raj_149> sipa: does that then also imply collisons are much common in case of incremental hashing? or its collison is not related to incremeental property atall?
10:11 < uproar> Use case, hmm maybe: produce new soundness proofs based on the UTXO set?
10:12 < sipa> uproar: muhash/ecmh/... don't permit compact proofs of inclusion or exclusion, so no - all you can do is check easily whether two sets you already have are identical
10:12 < nehan> it can also serve as a checksum to check for corruption of the utxoset
10:12 < sipa> raj_149: i don't know how to answer that
10:12 < fjahr> How about total_amount? What can that be used for? :)
10:13 < sipa> raj_149: it turns out that it's much harder to construct a homomorphic set hash and have it still be collision resistant
10:13 < fjahr> total_amount is one of the values returned from gettxoutsetinfo
10:13 < uproar> fast auditing on total supply
10:14 < raj_149> fjahr: isn't that the total circulating supply?
10:14 < sipa> willcl_ark: i don't think this is useful for SPV wallets (unless they also maintain a full UTXO set, which would be kind of pointless...)
10:14 < uproar> raj_149 it's the total sum of UTXO maybe not circulating nor circulatable
10:14 < fjahr> uproar: raj_149: yes! and that was one of the motivations that auditing could actually be really fast using the index :)
10:15 < raj_149> fjahr: i think its very useful then to get normies get into Bitcoin. Thats the golden number to show. :D
10:16 < uproar> fjahr fast amount/supply auditing is something I'd like but on the other hand I question to what end does it serve nodes (rather than users of those nodes), is MuHash decreasing of node-operation costs when the command is run?
10:16 < fjahr> Next question: did you come across downsides from implementing muhash for gettxoutset?
10:17 < raj_149> uproar: yes right, thanks for pointing that.
10:17 < willcl_ark> sipa: hmmm yes, I guess you get nothing more from this as an SPV node than querying multiple full nodes for headers
10:17 -!- Prayank [c6fc991c@198.252.153.28] has joined #bitcoin-core-pr-reviews     
10:17 < sipa> uproar: muhash itself is slower than the current hash, but the advantage is that it can be incrementally computed and maintained in the background (updating the hash at every block, rather than at RPC time)... making the RPC instantaneous
10:17 < fjahr> raj_149: yeah, and it's definitely much cooler if it takes a second rather than minutes :)
10:17 < uproar> the UX benefit is: look no surprising inflation! but from the node operation perspective is: There shouldn't have been unless some pretty fundamental aspect about validation are broken
10:18 < adiabat> how much slower is it if you're only running once after full IBD?
10:18 < uproar> sipa that's a valuable datum
10:18 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has joined #bitcoin-core-pr-reviews     
10:18 < sipa> adiabat: my very rough guess is 5x or so
10:18 < sipa> (of course, if you're I/O bottlenecked it's much less)
10:18 < adiabat> oh that's not too bad.  Once you run it more than a few times you're ahead
10:18 < uproar> gettxoutsetinfo is painfully slow even on high resource machines
10:19 < uproar> currently, that is >:]
10:19 < sipa> with this PR it becomes a lot worse
10:20 < sipa> fjahr: i guess you have more recent performance numbers
10:20 < raj_149> sipa: i dont understand, how is it worse?
10:20 < sipa> raj_149: because MuHash is many times slower than SHA256
10:20 < fjahr> it definitely exceedes the standard rpc timeout on lower powered machines, which was the main driver to get concerned for me.
10:20 < sipa> (and the current hash uses SHA256)
10:21 < willcl_ark> Ok so it's slower, but happens in the background, at each new block?
10:21 < raj_149> sipa: but the node will do that hashing internally for each block right? so getutxosetinfo will be very fast, right?
10:21 < sipa> willcl_ark: not with this PR
10:21 < nehan> raj_149: this change doesn't actually implement a rolling UTXOset hash. It just uses MuHash to calculate it when requested
10:21 < willcl_ark> ah ok
10:21 < sipa> willcl_ark: this just swaps out the SHA256 based hash for MuHash
10:21 < sipa> but it paves the way for a background-updated utxo set hash
10:21 < uproar> sipa is it that the first gettxoutsetinfo is going to get substantially slower but the incremental calls on gettxoutsetinfo will get faster relative to old gettxoutsetinfo subsequent calls?
10:21 < willcl_ark> 1 step back...
10:21 < sipa> uproar: no, all of them
10:22 < raj_149> nehan: ah ok.. thats true.
10:22 < adiabat> I guess for future PRs there are several options: increment at each block, or update a cache every time gettxouset is called and increment then
10:22 < fjahr> I don't have them at hand right now but I think I posted them in an older pr version
10:22 < sipa> uproar: again, this PR doesn't do any background hashing
10:22 < willcl_ark> we need the next PR to make it rolling
10:22 < adiabat> sipa: is later intent to make it happen every block or only update when requested? (or something else?_
10:23 < sipa> adiabat: ask fjahr :)
10:23 < jnewbery> it also adds a flag so you can still get the legacy SHA256 hash from gettxoutset if you want
10:23 < willcl_ark> Would it not be wise to add a commit for the rolling hash as part of this PR? Why would you split this way (with a performance degredation)?
10:23 < raj_149> basic question: utxo set is updated currently at each new block? or by some other trigger?
10:23 < michaelfolkson> There's another downside in terms of time taken to brute force the pre-image right? Once the pre-image is known from the original hash an attacker can brute force the pre-image for the new hash in a shorter time than normal?
10:23 < fjahr> will_clark: yes, that was 18000 which had everything, now i have split it up and this is the first part.
10:23 < adiabat> fjahr: is the intent.. :)
10:23 < willcl_ark> fjahr: I see
10:24 < sipa> michaelfolkson: it is supposed to have 128-bit collision resistance security, just like SHA256
10:24 < fjahr> adiabat: The plan is to update with every block/reorg and have an index
10:24 < sipa> adiabat: updating only when requested requires iterating over the whole utxo still, so that wouldn't give any gain
10:24 < fjahr> so older states can be queried quickly as well
10:25 < michaelfolkson> When you say "supposed to" sipa what does that mean? Makes it sound like you doubt that claim ;)
10:25 -!- Talkless [~Talkless@hst-227-49.splius.lt] has joined #bitcoin-core-pr-reviews     
10:26 < fjahr> Keep those questions coming :) But here is one of mine as well: This PR not only adds Muhash but also TruncatedSHA256Writer. Why?
10:26 < adiabat> I don't see why updating when requested needs to look at the whole utxo set?  Couldn't it replay blocks since last call and perform the additions / deletions that way?
10:26 < fjahr> TruncatedSHA512Writer actually :D
10:26 < jnewbery> michaelfolkson: that's because it's a hash function. We assume that it has that much security until someone breaks it
10:27 < adiabat> (though I guess that could be even slower if it's very infrequently called)
10:27 < raj_149> fjahr: wild guess. to add extra layer of security?
10:27 < sipa> michaelfolkson: assuming the discrete logarithm in GF(2^3072 - 1103717) takes is 128-bit secure, that ChaCha20 and truncated SHA512 have their intended security, ...
10:27 < jnewbery> adiabat: we don't have those utxos any more after the block has been connected
10:27 < raj_149> and may be added collison resistance..
10:27 < michaelfolkson> Ok gotcha thanks sipa jnewbery
10:27 < sipa> adiabat: oh sure, it could, but that'd be incompatible with pruning
10:28 < uproar> maybe I'm missing something but if currently SHA256 hasing in gettxoutsetinfo makes calls take ~5 minutes, and MuHash increases the time, what's the benefit? being able to do it in the background or that the total sum of updates costs less because of the "homomorphic set" properties?
10:28 < adiabat> ah right that makes sense
10:28 < sipa> uproar: updating is super fast
10:28 < nehan> sipa: can your replay blocks to update the rolling utxoset hash without the utxoset at the point you start replaying?
10:28 < sipa> nehan: sure
10:28 < jnewbery> right, you could look up each utxo in the previous block files, but that would be painfully slow, and incompatible with pruning
10:28 < sipa> if you still have the block
10:29 < sipa> and the undo data
10:29 < uproar> sipa ok, thanks I think i mistook "Muhash is slower" to mean "the entire update process for post PR will be slower" which I take be not the case
10:29 < nehan> sipa: are you suggesting you'd have to reconstruct the UTXOset at that point in the past using the undo data?
10:29 < sipa> uproar: yes it will be much slower, but it paves the way for incrementally computing the hash, which will make the RPC instant
10:29 < jnewbery> oh sorry I'm wrong - of couse you can use the undo data
10:29 < sipa> uproar: however this PR doesn't do any of the incrementality
10:29 < uproar> now I understand the bigger picture, thanks
10:30 -!- r251d [~r251d@162-196-59-192.lightspeed.irvnca.sbcglobal.net] has quit [Quit: r251d]     
10:30 < sipa> nehan: no
10:30 < fjahr> ray_149: so the truncated hash does not decrease security but i don't think it increases it either. It is an efficient way to turn the raw data of utxos into a 256bit number.
10:30 < nehan> sipa: k, thanks
10:30 -!- MajkiMajk [57ce3181@87-206-49-129.dynamic.chello.pl] has quit [Remote host closed the connection]     
10:31 < raj_149> fjahr: but why trucncated 512? why not sha256 it directly?
10:31 < sipa> nehan: blocks are "patches" to the UTXO set; if you have the hash state as of block N, you you roll that state forward by applying the blocks after it to it
10:31 < sipa> you don't need the actual UTXO set anymore at that point
10:31 < fjahr> well, sipa made the choice but from the research papers I saw it is faster
10:31 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has quit [Quit: The Lounge - https://thelounge.chat]     
10:32 < willcl_ark> faster to SH512 and then truncate?
10:32 < sipa> yeah, it's because most UTXOs are of a size that would require 2 SHA256 compressions, but only one SHA512 compression
10:32 < sipa> so SHA512 may be faster for such inputs
10:32 -!- Landryl [~Landryl@ns528256.ip-192-99-10.net] has joined #bitcoin-core-pr-reviews     
10:32 < sipa> it's probably worth benchmarking that again, now that we have AVX2 and SHA-NI optimized SHA256
10:32 < fjahr> There is a link to a SHA512/256 research paper somewhere in a #18000 comment
10:33 < fjahr> I think that was the best resource I found if you want to dig deeper
10:33 < fjahr> https://eprint.iacr.org/2010/548.pdf
10:34 < fjahr> found it :)
10:34 < willcl_ark> StackExchange is tellign me the max message size for SHA256 is 2 million terabytes
10:34 < willcl_ark> thanks fjahr
10:34 < fjahr> sipa: correct me if that is not the right resource
10:34 < sipa> willcl_ark: that's correct, but i don't think that's relevant?
10:34 < sipa> fjahr: i think the right resource is benchmarking yourself :)
10:34 < fjahr> :)     
10:35 < sipa> UTXOs are limited to 10000 bytes ish
10:35 < willcl_ark> sipa what is " most UTXOs are of a size that would require 2 SHA256 compressions" in reference to then?
10:35 < uproar> I think it's indicating the mechanics of how 256 blocks are compressed
10:35 < sipa> willcl_ark: SHA256 transforms an internal 32-byte state by consuming 64 byte blocks at a time; the runtime is proportional to have many of those 64-byte blocks you need to consume
10:36 < uproar> max message size and how many blocks you have to compress aren't proportional
10:36 < fjahr> Ok, I hope it's not too obvious for everyone, but: Why is the Muhash class implemented in Bitcoin Core and not in libsecp256k1?
10:36 < willcl_ark> hmmm ok. that's definitely a good one for me to read up on :)
10:36 < sipa> SHA512 transforms an internal 64-byte state by consuming 128 byte blocks at a time; the runtime per consumption is longer than SHA256's, but typically faster per byte
10:36 < sipa> willcl_ark: as UTXOs are often larger than 64 bytes, but less than 128 bytes, they'd need 2 SHA256 consumptions, or 1 SHA512 consumption
10:36 < sipa> so the question is which of those two is faster
10:37 < raj_149> sipa: correct me if i am wrong, but its are not hashing the utxos directly, its hashing the finalised muhash, which is 3072 bit.
10:37 < willcl_ark> sipa: ah ok. that makes it clear now!
10:37 < fjahr> or why would ECMH be in lipsecp and not Muhash/
10:37 < fjahr> ?     
10:37 < sipa> raj_149: no, the individual UTXOs are hashed, to produce a 256-bit key; that key is than expanded into a 3072-bit number using ChaCha20
10:38 < sipa> the 3072-bit numbers are then multiplied modulo 2^3072-1103717
10:38 < sipa> and then the final 3072-bit output is again SHA512 hashed to produce a 256-bit output
10:38 < sipa> *then
10:38 < jonatack> hi     
10:39 < fjahr> hi jonatack
10:39 < jonatack> great PR, but separated from 18000 it's a bit neither here nor there ;)
10:39 < jonatack> i'll explain:
10:39 < jnewbery> the final 3072-bit output could be hashed using SHA256, but I suppose you use 512 for consistency?
10:39 < sipa> jnewbery: yeah
10:39 < sipa> that one doesn't matter
10:39 -!- shaunsun [shaunsun@gateway/vpn/privateinternetaccess/shaunsun] has quit [Ping timeout: 240 seconds]     
10:40 < raj_149> sipa: ok got it. i thought we were talking about the final one here.
10:40 < jonatack> fjahr: it is good that you added the legacy_hash bool to the RPC, but i would suggest making MuHash opt-in rather than the default until the index is merged
10:40 < fjahr> well, I hope most know libsecp does eliptic curve crypto and there is not EC used in muhash but ECMH does use it :)
10:40 < jnewbery> currently SHA512 is only used for encrypting the wallet and seeding the PRNG as far as I can tell
10:40 < sipa> fjahr: if you're going to cache the hash for every block... that's actually an argument in favor of ECMH, as the minimal "state" to keep for ECMH is 33 bytes only, while for MuHash it's 384 bytes
10:41 < jonatack> because otherwise gettxoutsetinfo is essentially broken for me on testnet and mainnet until then
10:41 < uproar> willcl_ark this is a nice (slowish) breakdown of how sha 256 works.  see the video of it verbally annotated https://github.com/in3rsha/sha256-animation
10:41 < fjahr> sipa; I am only writing the finalized hash into the index otherwise there is one muhash that is update with every block
10:42 < willcl_ark> thanks uproar, I will take a look at that later
10:42 < sipa> fjahr: ah, gotcha
10:42 < sipa> that makes sense
10:42 < jnewbery> jonatack: by broken do you mean 'i have to pass a flag'?
10:42 < fjahr> jonatack: you mean it's too slow, right?
10:42 < thomasb06> jonatack: as expected, the archwiki admins are not interested to make a new page on how to compile and test the Bitcoin core. Your page will remain the reference for Linux builds for another while...
10:43 < jonatack> jnewbery: yes, without the flag it times out and raises... i haven't found the counterflag for that yet :)
10:43 < jnewbery> thomasb06: that's off-topic. Perhaps you can message jonatack after the meeting
10:43 < jonatack> fjahr: right
10:43 < willcl_ark> jonatack: I think I'd agree that defaulting to the faster impl. makes sense, with a flag for the new.
10:43 < fjahr> yeah, I am still a bit undecided what the right way to go is concerning the old way, do we want it temporary or permanently
10:44 < jnewbery> jonatack: I agree that the default should be switched to maintain existing behaviour
10:44 < jonatack> i'm building the index now... ~100k blocks/hour to build so far
10:44 < jonatack> (with #18000)
10:45 < jonatack> jnewbery: yes
10:45 < willcl_ark> I didn't try the PR on my server, but current performance is 60 seconds exactly for gettxoutset so at ~5x slower that will be an issue (that a flag can solve, but still...)
10:45 < nehan> fjahr: at the very least don't enable it by default until you have the index, since this change just makes gettxoutsetinfo strictly worse
10:45 < jonatack> nehan: ^
10:45 -!- shaunsun [shaunsun@gateway/vpn/privateinternetaccess/shaunsun] has joined #bitcoin-core-pr-reviews     
10:46 < fjahr> nehan: for the people who don't run the index it will always be worse, so I guess it would need to stay the default then :)
10:46 < uproar> nehan is your comment about optimal migration or something else I didn't understand?
10:47 < fjahr> noted that default should be the old one
10:47 < nehan> uproar: i think migration? not sure what you mean. but yeah, whether to turn something on by default and in what order to do so
10:47 < uproar> e.g. "keep using the old method until the new method has caught up" or something else
10:47 < nehan> fjahr: hmm that's tricky. and if it's off by default you've got two hashes for the utxoset in the world.
10:47 < jnewbery> fjahr: I don't agree. Once there's an optional index, then I think it's possible we might want to remove the SHA256 version
10:48 < sipa> agree
10:48 < sipa> maybe not immediately, but certainly with a deprecation window that should be ok
10:48 -!- Caralie [2fd149de@47-209-73-222.lkchcmtk01.res.dyn.suddenlink.net] has joined #bitcoin-core-pr-reviews     
10:48 < jonatack> yes     
10:48 < nehan> jnewbery: sipa: how did you reach that conclusion? cause the rpc is not used very much?
10:48 < jnewbery> you have two options: run an index and have fast access to the utxo hash, or don't and be a little bit patient
10:48 < fjahr> hm, ok, I will have to think about it and test more on slower machines
10:49 < sipa> nehan: i suspect that once the index code exist, everyone who more than very exceptionally uses gettxoutsetinfo will enable the index (as it's pretty much free)
10:49 < willcl_ark> is the RPC timeout 2 minutes?
10:49 < jnewbery> if you're someone who wants to query the utxo set hash frequently, then build the index. If you're not, a one-off hit on the rare ocassion you do run it is acceptable
10:49 < fjahr> 15 is the default i think
10:49 < willcl_ark> oh, right
10:50 < nehan> jonatack: ah. i see i just restated what you said earlier with the comment on the default flag
10:51 < fjahr> Ok, I have another question on hashing algos but I would skip that unless someone has thoughts on it.
10:51 < thomasb06> (do you have the doc page for RPC under the arm by the way?)
10:51 < fjahr> For those that tested the code already: What are your thoughts on the Muhash benchmarks, e.g. (a) the benchmarking code, and (b) your observations if you tested the code locally?
10:51 < jonatack> nehan: i was seconding your comment :)
10:52 < nehan> fjahr: depends onif anyone is relying on it. how do people usually use this rpc?
10:52 -!- pinheadm_ [~pinheadmz@pool-100-33-69-78.nycmny.fios.verizon.net] has joined #bitcoin-core-pr-reviews     
10:52 < sipa> tbh, i mostly use it for statistics like number of UTXOs and circulating supply
10:52 < uproar> same
10:53 < sipa> but once there is a fast way to compute a utxo set hash (or access it), i think that aspect of it will become much more useful
10:53 < nehan> if no one is relying on the old hash format it seems quite reasonable to get rid of it!
10:54 < sipa> we generally have a policy of not breaking RPC compatibility without deprecation cycle
10:54 < jnewbery> fjahr: my high-level thought about the PR is that it's good that you've split this from #18000, but there's still a lot of code there to review. It's a mix of python cryptography, C++ crypto, ASM, RPC code,... If you could split off smaller parts to review, it'd be easier to make progress
10:54 < jonatack> same, and #18000 is very welcome to me because it's that RPC is nigh unuseable at the moment
10:54 < fjahr> The two use cases I met mostly were: a) "for fun" to check if the node was running and to se the stats and b) checking circulating supply regularly (Bitmex publishes the numbers for example)
10:54 -!- pinheadmz [~pinheadmz@96.47.229.196] has quit [Ping timeout: 264 seconds]     
10:54 < jnewbery> (that assumes that there is consensus ACK for the overall concept and approach)
10:54 < fjahr> jnewbery: yeah, it has grown the last two days, agree
10:55 < sipa> fjahr: one thing that could be done if you have the index is make the startup consistency check roll back the utxo hash a few blocks, and compare it with the index
10:55 < sipa> which would prove correctness of the block and undo data on disk
10:56 < jonatack> jnewbery: fjahr: i agree, it's a long and hard review because it's so wide ... multi-day if you get into the crypto algo implementation and the assembly optimising
10:56 < fjahr> sipa: interesting, will conisder adding it when adding the index
10:57 < jonatack> nice
10:57 < fjahr> ok, three minutes to go, it went way too fast as always. Any last comments?
10:57 < fjahr> or questions?
10:58 < fjahr> I am always open for DMs if you have questions on this btw
10:58 < sipa> fjahr: thanks for taking this up, and helping push it forward :)
10:58 < willcl_ark> yes thanks fjahr, it's a nice PR IMO, once it's fully-formed
10:58 < uproar> much thanks fjahr !
10:58 -!- Caralie [2fd149de@47-209-73-222.lkchcmtk01.res.dyn.suddenlink.net] has quit [Remote host closed the connection]     
10:59 < jnewbery> If there's a split PR on the python implementation, it'd be great to have a review club on that where we can really get into the weeds on the crypto :)
10:59 < nehan> thanks fjahr!
10:59 < raj_149> thanks fjahr for hosting the review sesion too
10:59 < troygiorshev> thanks fjahr
10:59 < jonatack> thanks fjahr and sipa!
10:59 < raj_149> jnewbery: that sounds very fun.
10:59 < uproar> jnewbery +!
10:59 < uproar> +1*     
11:00 < fjahr> Thanks everyone for attending, especially sipa :)
11:00 < jnewbery> thanks fjahr!
11:00 < fjahr> #endmeeting
11:00 < thomasb06> thanks fjahr
11:00 < theStack> thanks for hosting fjahr, that was an interesting read today
11:00 < sipa> jnewbery: it's 80 lines, how complex can it be? *hides*
11:00 < uproar> thanks sipa fjahr
11:01 -!- Prayank [c6fc991c@198.252.153.28] has quit [Ping timeout: 245 seconds]     
11:01 -!- tarboss [~tarboss@p54a034ab.dip0.t-ipconnect.de] has left #bitcoin-core-pr-reviews []     
11:02 < raj_149> for some reason i find c++ way easier to read than python. am i the only one? or its just a noob symptom? :D
11:02 < sipa> raj_149: it may be that you're just far more familiar with c++
11:03 < sipa> raj_149: but it may also be the case that there are details in the c++ code that you're missing that you don't even realize you're missing
11:03 < jnewbery> sipa: I bet you could do it in 50 if you tried a bit harder :)
11:03 -!- ecurrencyhodler [68ac287f@cpe-104-172-40-127.socal.res.rr.com] has quit [Remote host closed the connection]     
11:03 < sipa> jnewbery: fine, i'll remove the comments and write everything using reduce filters
11:03 < thomasb06> here: https://en.wikipedia.org/wiki/Remote_procedure_call
11:04 < uproar> I heard chuck Norris did a few algos in negative amount of lines of code
11:05 < raj_149> sipa: not even remotely, i started learnig with pyhton, c++ is newer to me. probably the typed nature of the language helps. in python i always find it difficult to see structure of objects, unless i write it myself. may be i am missing some key elements of reading python.
11:05 < sipa> raj_149: i think the bigger picture of the set hash (which things are hashed, combined, stitched together, ...) is far more obvious in the python code
11:06 < sipa> uproar: he can of course also recite the digits of pi backwards
11:06 < raj_149> sipa: where is the python implemetation lying? can i take a look?
11:06 < sipa> fjahr added it to the PR
11:06 < uproar> sipa lool
11:06 < sipa> https://github.com/bitcoin/bitcoin/pull/19055/commits/4438aed09e87de0afb37da64ffb5e7489084e8ab
11:07 -!- gzhao408 [49fcfb03@c-73-252-251-3.hsd1.ca.comcast.net] has quit [Remote host closed the connection]     
11:08 < raj_149> sipa: found it, it would fun to run a debugger over it. :D will do that by the weekend.
11:11 -!- danielyin [~danielyin@204.11.107.220] has joined #bitcoin-core-pr-reviews     
11:34 -!- dfmb_ [~dfmb_@unaffiliated/dfmb/x-4009105] has quit [Quit: Leaving]     
11:38 -!- theStack [~honeybadg@vps1648322.vs.webtropia-customer.com] has quit [Quit: Lost terminal]     
11:57 -!- uproar [~uproar@178.162.222.42.adsl.inet-telecom.org] has left #bitcoin-core-pr-reviews []     
12:24 -!- Talkless [~Talkless@hst-227-49.splius.lt] has quit [Quit: Konversation terminated!]     
12:34 -!- tryphe [~tryphe@unaffiliated/tryphe] has quit [Read error: Connection reset by peer]     
12:35 -!- tryphe [~tryphe@unaffiliated/tryphe] has joined #bitcoin-core-pr-reviews     
12:48 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has quit [Remote host closed the connection]     
12:55 -!- troygiorshev [~troygiors@CPEdcef09a0ed55-CM0c473d74be00.cpe.net.cable.rogers.com] has quit [Quit: Lost terminal]     
13:03 -!- troygiorshev [~troygiors@CPEdcef09a0ed55-CM0c473d74be00.cpe.net.cable.rogers.com] has joined #bitcoin-core-pr-reviews     
13:10 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has joined #bitcoin-core-pr-reviews     
13:10 -!- thomasb06 [~user@eth-west-pareq2-46-193-0-224.wb.wifirst.net] has quit [Remote host closed the connection]     
13:26 -!- troygiorshev [~troygiors@CPEdcef09a0ed55-CM0c473d74be00.cpe.net.cable.rogers.com] has quit [Quit: Lost terminal]     
13:27 -!- jonatack [~jon@184.75.221.43] has quit [Ping timeout: 246 seconds]     
13:29 -!- jonatack [~jon@37.166.127.107] has joined #bitcoin-core-pr-reviews     
13:48 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Remote host closed the connection]     
13:48 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #bitcoin-core-pr-reviews     
14:25 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Remote host closed the connection]     
14:25 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #bitcoin-core-pr-reviews     
14:35 -!- pinheadm_ [~pinheadmz@pool-100-33-69-78.nycmny.fios.verizon.net] has quit [Quit: pinheadm_]     
14:36 -!- pinheadmz [~pinheadmz@pool-100-33-69-78.nycmny.fios.verizon.net] has joined #bitcoin-core-pr-reviews     
16:00 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Remote host closed the connection]     
16:00 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #bitcoin-core-pr-reviews     
16:09 -!- jonatack_ [~jon@37.172.61.28] has joined #bitcoin-core-pr-reviews     
16:12 -!- jonatack [~jon@37.166.127.107] has quit [Ping timeout: 260 seconds]     
16:14 -!- shaunsun [shaunsun@gateway/vpn/privateinternetaccess/shaunsun] has quit [Ping timeout: 272 seconds]     
16:32 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has quit [Remote host closed the connection]     
17:00 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Remote host closed the connection]     
17:01 -!- pinheadmz [~pinheadmz@pool-100-33-69-78.nycmny.fios.verizon.net] has quit [Ping timeout: 260 seconds]     
17:01 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #bitcoin-core-pr-reviews     
17:20 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Read error: Connection reset by peer]     
17:21 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-core-pr-reviews     
18:33 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has joined #bitcoin-core-pr-reviews     
18:38 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has quit [Ping timeout: 264 seconds]     
18:41 -!- pinheadmz [~pinheadmz@pool-100-33-69-78.nycmny.fios.verizon.net] has joined #bitcoin-core-pr-reviews     
19:40 -!- shesek [~shesek@unaffiliated/shesek] has quit [Read error: Connection reset by peer]     
19:40 -!- shesek [~shesek@185.3.145.28] has joined #bitcoin-core-pr-reviews     
19:40 -!- shesek [~shesek@185.3.145.28] has quit [Changing host]     
19:40 -!- shesek [~shesek@unaffiliated/shesek] has joined #bitcoin-core-pr-reviews     
21:16 -!- instagibbs [~instagibb@pool-71-178-191-230.washdc.fios.verizon.net] has quit [Ping timeout: 272 seconds]     
21:23 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]     
21:30 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-pr-reviews     
22:00 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Quit: Leaving]     
22:00 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #bitcoin-core-pr-reviews     
22:33 -!- Mittie27Mraz [~Mittie27M@static.57.1.216.95.clients.your-server.de] has joined #bitcoin-core-pr-reviews     
22:39 -!- Mittie27Mraz [~Mittie27M@static.57.1.216.95.clients.your-server.de] has quit [Ping timeout: 264 seconds]     
23:06 -!- danielyin [~danielyin@204.11.107.220] has quit [Quit: leaving]     
23:52 -!- PaulTroon [~paultroon@h-5-150-248-150.NA.cust.bahnhof.se] has joined #bitcoin-core-pr-reviews