--- Log opened Wed Oct 20 00:00:41 2021 --- Log closed Wed Oct 20 05:56:31 2021 --- Log opened Wed Oct 20 05:57:09 2021 05:57 -!- gnusha [~gnusha@user/gnusha] has joined #bitcoin-core-pr-reviews 05:57 -!- Topic for #bitcoin-core-pr-reviews: Meeting every Wednesday at 17:00 | See https://bitcoincore.reviews/ for details | This channel is logged 05:57 -!- Topic set by jnewbery [] [Mon May 24 09:24:20 2021] 05:57 [Users #bitcoin-core-pr-reviews] 05:57 [ _0x0ff ] [ dergoegge ] [ harding ] [ koolazer ] [ notmandatory ] [ schmidty ] 05:57 [ _aj_ ] [ dhruv ] [ hebasto ] [ laanwj ] [ pg2 ] [ shiza ] 05:57 [ achow101 ] [ djinni`_ ] [ Henry151 ] [ larryruane ] [ pinheadmz ] [ sipa ] 05:57 [ ajonas ] [ dongcarl ] [ hugohn ] [ lightlike ] [ promag_ ] [ siv2r[m] ] 05:57 [ amiti ] [ dr_orlovsky] [ jarolrod ] [ livestradamus ] [ provoostenator] [ stick ] 05:57 [ Amnesia ] [ emzy ] [ jeremyrubin ] [ luke-jr ] [ qubenix ] [ stick[m] ] 05:57 [ avril ] [ fanquake ] [ jkczyz ] [ MarcoFalke ] [ r-ush ] [ takinbo ] 05:57 [ belcher ] [ fdov ] [ jnewbery ] [ MatrixBot1234510] [ raj ] [ TheRec ] 05:57 [ blkncd ] [ FelixWeis ] [ johnzweng ] [ mdrollette ] [ RubenSomsen ] [ theStack ] 05:57 [ commmon ] [ fjahr ] [ jomat ] [ merkle_noob[m] ] [ ryanofsky ] [ vasanth2[m]] 05:57 [ Cory ] [ ghost43 ] [ jonasschnelli] [ meshcollider ] [ S3RK ] [ waxwing_ ] 05:57 [ David[m]12345] [ glix_ ] [ josibake ] [ michaelfolkson ] [ SandipanDey[m]] [ willcl_ark ] 05:57 [ DavidBakin ] [ glozow ] [ jrawsthorne_ ] [ Murch[m] ] [ sandipndev ] [ yakshaver ] 05:57 [ davterra ] [ gnusha ] [ kanzure ] [ nathanael ] [ sanket1729_ ] [ yonson ] 05:57 [ DeanGuss ] [ greypw254 ] [ kcalvinalvin ] [ neha ] [ sanket_cell_ ] [ Zenton ] 05:57 -!- Irssi: #bitcoin-core-pr-reviews: Total of 90 nicks [0 ops, 0 halfops, 0 voices, 90 normal] 05:58 -!- Channel #bitcoin-core-pr-reviews created Wed May 19 12:43:59 2021 06:00 -!- Irssi: Join to #bitcoin-core-pr-reviews was synced in 180 secs 06:12 -!- r-ush [~r-ush@52.187.184.81] has quit [Ping timeout: 250 seconds] 06:14 -!- r-ush [~r-ush@52.187.184.81] has joined #bitcoin-core-pr-reviews 07:15 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has joined #bitcoin-core-pr-reviews 07:20 -!- pg2 [sid518209@id-518209.hampstead.irccloud.com] has quit [Ping timeout: 245 seconds] 07:22 -!- josibake [sid509132@id-509132.helmsley.irccloud.com] has quit [Ping timeout: 258 seconds] 07:23 -!- josibake [sid509132@helmsley.irccloud.com] has joined #bitcoin-core-pr-reviews 07:35 -!- pg2 [sid518209@id-518209.hampstead.irccloud.com] has joined #bitcoin-core-pr-reviews 07:46 -!- dunxen [~dunxen@gateway/tor-sasl/dunxen] has joined #bitcoin-core-pr-reviews 08:09 -!- waxwing_ is now known as waxwing 08:09 -!- waxwing [~waxwing@193.29.57.116] has quit [Changing host] 08:09 -!- waxwing [~waxwing@user/waxwing] has joined #bitcoin-core-pr-reviews 08:20 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has quit [Ping timeout: 260 seconds] 08:22 -!- varioust [~varioust@72-46-48-28.lnk.ne.static.allophone.net] has joined #bitcoin-core-pr-reviews 09:02 -!- varioust [~varioust@72-46-48-28.lnk.ne.static.allophone.net] has quit [Ping timeout: 264 seconds] 09:05 -!- varioust [~varioust@72-46-48-28.lnk.ne.static.allophone.net] has joined #bitcoin-core-pr-reviews --- Log closed Wed Oct 20 09:14:18 2021 --- Log opened Wed Oct 20 09:14:18 2021 09:25 -!- Talkless [~Talkless@mail.dargis.net] has joined #bitcoin-core-pr-reviews 09:34 -!- Naiza [~Naiza@180.188.251.235] has joined #bitcoin-core-pr-reviews 09:40 -!- jamesob [uid180710@id-180710.helmsley.irccloud.com] has joined #bitcoin-core-pr-reviews 09:42 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has joined #bitcoin-core-pr-reviews 09:42 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has quit [Remote host closed the connection] 09:42 -!- esraa [~esraa@147.236.159.129] has joined #bitcoin-core-pr-reviews 09:42 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has joined #bitcoin-core-pr-reviews 09:46 -!- tr3xx [~tr3xx@2a01:799:5f:ec00:d9b9:6729:9569:c6ea] has joined #bitcoin-core-pr-reviews 09:47 -!- Kaizen_Kintsugi7 [~Kaizen_Ki@d137-186-173-66.abhsia.telus.net] has joined #bitcoin-core-pr-reviews 09:51 -!- Kaizen_K_ [~Kaizen_Ki@node-1w7jr9yi65te55mxt1kkm1n0k.ipv6.telus.net] has joined #bitcoin-core-pr-reviews 09:51 -!- Kaizen_Kintsugi7 [~Kaizen_Ki@d137-186-173-66.abhsia.telus.net] has quit [Client Quit] 09:52 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has quit [Ping timeout: 252 seconds] 09:54 < Kaizen_K_> isn't #bitcoin-core-dev on irc.libera? 09:57 < sipa> yes 09:58 < sipa> you were there, but left the channel around 15 minutes ago 09:58 -!- svav [~svav@82-69-86-143.dsl.in-addr.zen.co.uk] has joined #bitcoin-core-pr-reviews 09:58 < Kaizen_K_> hmm 09:59 < Kaizen_K_> uhg! not used to this irc client, thx for the heads up 10:00 < dergoegge> #startmeeting 10:00 < dergoegge> Hi everyone! Welcome to this week's PR Review Club! 10:00 -!- ajayparmar904 [~ajayparma@103.39.129.252] has joined #bitcoin-core-pr-reviews 10:00 < jnewbery> hi! 10:00 < glozow> hi! 10:00 < esraa> hello 10:00 < dergoegge> feel free to say hi to let people know you are here 10:01 < svav> Hi! 10:01 < ajayparmar904> Hi 10:01 < dergoegge> is anyone here for the first time? :) 10:01 < tr3xx> hi! 10:01 < larryruane> hi 10:01 < ajayparmar904> Yes.. i am here for first time 10:01 < esraa> first time here (: 10:01 < Kaizen_K_> 2nd time, just trying to build a habit 10:01 < Kaizen_K_> hi all 10:01 < dergoegge> ajayparmar904: welcome! 10:01 < dergoegge> esraa: hi, welcome! 10:01 < svav> New all time high today! 10:02 < tr3xx> dergoegge: third time for me :) 10:02 < dergoegge> today we are looking at #17631 - Expose block filters over REST 10:02 < jnewbery> ajayparmar904 esraa: welcome! 10:02 < David[m]12345> 2nd time and no clue, just lurking, thx :) 10:02 -!- stickies-v [~stickies-@84-45-243-170.static.enta.net] has joined #bitcoin-core-pr-reviews 10:02 < dergoegge> notes and questions are in the usual place: https://bitcoincore.reviews/17631 10:02 < dergoegge> lurkers are also welcome! 10:02 -!- Azorcode [~Azorcode@201.242.162.105] has joined #bitcoin-core-pr-reviews 10:02 < stickies-v> hi - sorry I'm late! 10:02 < dergoegge> ok lets get started: Did you review the PR? Concept ACK, approach ACK, tested ACK, or NACK? 10:03 < stickies-v> approach NACK 10:03 < svav> I read the notes ... 10:03 < jnewbery> concept & approach ACK. I'd like Matt to fix his commit log before I ACK it though :) 10:04 < sipa> concept ACK 10:04 < dergoegge> jnewbery: oh i pointed that out for the PR description, did not see it for the commit 10:05 < Kaizen_K_> what does ACK mean? 10:05 < larryruane> utACK 10:05 -!- ___nick___ [~quassel@cpc68286-cdif17-2-0-cust533.5-1.cable.virginm.net] has joined #bitcoin-core-pr-reviews 10:05 < glozow> Kaizen_K_: https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#peer-review 10:06 < jnewbery> dergoegge: ah. If you open a PR with just one commit, then github will use the commit log as the PR description by default. I guess that's where it came from. 10:06 < Kaizen_K_> glozow: ty 10:06 < dergoegge> ok next question: What are blockfilters and what are they used for? (hint: see BIP157 and BIP158) 10:06 < jnewbery> I think it's a shame that the github review process de-emphasizes commit logs. It'd be nice to be able to comment on them inline just like for the code. They're important! 10:07 < michaelfolkson> hhi 10:07 < svav> Does ACK stand for acknowledge or is it an acronym? 10:07 < dergoegge> jnewbery: thats true 10:07 < Kaizen_K_> dergoegge: I understand it as something related to the bloom filter 10:08 < stickies-v> Blockfilters are a replacement to bloom filters that allow light nodes to significantly reduce bandwidth, storage and verification without sacrificing privacy like bloom filters did. A block filter is a compressed list of prevouts and UTXOs in a block 10:08 < Kaizen_K_> where it allows a smaller amout of data to be sent around the network and reconstructed on the other end? 10:08 < sipa> there is no reconstruction 10:08 < Kaizen_K_> ah, thx stickies-v 10:09 < dergoegge> Kaizen_K_ : they enable a similar use case as the bloom filters but they are actually a replacement for the bloom filters 10:09 < glozow> the light client requests the entire block if the filter indicates there's something they're interested in 10:09 < sipa> it's just a way to quickly test whether a block may contain transactions that are interesting or not 10:09 < svav> A blockfilter is a filter on the data in a block, allowing a compact representation of the data. 10:09 < dergoegge> stickies-v: correct! 10:09 < larryruane> I looked up bips 157, 158, and they are both in "Draft" status .... what does that mean? It seems they've been implemented 10:09 < sipa> larryruane: BIP statuses are neglected 10:10 < Kaizen_K_> got it. 10:10 < dergoegge> glozow, sipa: yes! 10:10 < sipa> svav: i wouldn't call it "representation"; they don't encode the full block - more like a fancy checksum, which allows you to quickly check whether the block may be interesting to you 10:10 < sipa> but the check may be wrong (you may think a block is interesting while it isn't) 10:10 < larryruane> "..may contain transactions that are interesting .." Also an important goal is to not leak information to the server (bitcoind node) about WHAT we (light client) are interested in, right? 10:11 < sipa> larryruane: indeed, that's the primary difference with BIP37 10:11 < jnewbery> they're similar to BIP37 bloom filters in as much as they're a probabilistic filter of set inclusion. However, they're different in that everyone uses the same filter for each block, so they don't need to be recalculated for every client 10:11 < dergoegge> one thing to note is that the filters come with false positives (not sure what the exact % is on those) 10:11 < stickies-v> sipa but the filter can only be wrong for false positives, but never false negatives right? 10:11 < dergoegge> larryruane: yes! 10:11 < larryruane> jnewbery: thanks, that helps a lot! 10:11 < sipa> with BIP37, the filtering was done on the server side (client gives filter of what they're interested in to server); with BIP157, it's done on the client side (the server gives filter about the block's contents to the client) 10:11 < sipa> stickies-v: correct 10:12 < stickies-v> ty 10:12 < sipa> there is also a technical difference, in that BIP37 uses a bloom filter, while BIP157 uses a golomb-coded filter; the difference between those is mostly an implementation detail (bloom is bigger but faster to update) 10:12 < Kaizen_K_> this seems like a privacy enhancement to the bloom filter. The server doesn't really know what the client is up to 10:13 -!- ___nick___ [~quassel@cpc68286-cdif17-2-0-cust533.5-1.cable.virginm.net] has quit [Ping timeout: 260 seconds] 10:13 < dergoegge> Kaizen_K_: yes, that was one design goal of the BIPs 10:13 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has joined #bitcoin-core-pr-reviews 10:13 < jnewbery> For anyone interested in learning about the Bitcoin Core implementation of block filters, we did a whole series of review clubs on them. Just look in https://bitcoincore.reviews/meetings/ for anything with "BIP 157" in the title 10:14 < dergoegge> There is a great short explanation of the BIPs on the optech website: https://bitcoinops.org/en/newsletters/2019/04/23/#footnotes 10:14 < Kaizen_K_> jnewbery:ty 10:14 < stickies-v> it's also great for scaling, because now each full node only has to calculate one filter per block, whereas previously it would have to spend additional resources for each bloom filter (which was unique per light client) 10:14 < Kaizen_K_> ty dergoegge 10:14 < larryruane> malicious servers can't make things appear to be present in the block that aren't (since client should always download block and check), but server could *withhold* items (from the filter), I think? 10:14 < sipa> stickies-v: indeed, BIp37 was effectively a hard-to-avoid DoS risk 10:14 < Kaizen_K_> damn, that is cool 10:15 < sipa> larryruane: withholding is indeed a problem; the real solution to that is having PoW commit to the filters... 10:15 < stickies-v> larryruane yes, but that's where block filter headers come into play. You should connect to multiple nodes, check if they all provide the same block filter headers (they commit to block filters), and if you see different filters amongst nodes investigate. You then also verify that the filter header matches the filter. So quite easy to catch attackers as long as you're onnected to one hoenst node 10:15 < jnewbery> there's also a summary here: https://bitcoin.stackexchange.com/questions/86231/whats-the-distinction-between-bip-157-and-bip-158-are-they-supported-by-bitcoi , but that was just me copy-pasting from the same optech description 10:16 < sipa> stickies-v: unfortunately, it does mean that one attacker peer is enough to force you into worst-case bandwidth usage (download all blocks) 10:16 < glozow> do we have multiple FilterTypes or is that just there for future extensibility? 10:16 < sipa> glozow: BIP158 initially defined two filter types; following review, one was dropped 10:17 < dergoegge> glozow: i was also wondering about that 10:17 < stickies-v> sipa true, but once established that he's an attacker you disconnect from that peer so it's only temporary? 10:17 < sipa> so it's just for future extensibility 10:17 < glozow> sipa: ah, thanks 10:17 < dergoegge> sipa: thanks 10:17 < larryruane> " ... PoW commit to the filters ..." That's cool, maybe that could be done in a future softfork? 10:17 < sipa> larryruane: yes, that's exactly what i mean 10:17 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has quit [Ping timeout: 264 seconds] 10:18 < dergoegge> Ok i think we covered blockfilters 10:18 < dergoegge> Next question: Can you explain what REST is? 10:18 < larryruane> very much an industry standard! 10:18 < dergoegge> My understanding of REST is very basic and i hope one of you has a great answer :D 10:18 < Kaizen_K_> I just understand it as an api access thtough the internet 10:19 < larryruane> tons and tons of "devices" (the kind I'm familiar with is storage nodes) can expose a REST interface to control and/or query the device 10:19 < jnewbery> There was lots of discussion on the mailing list about what to include in the block filters as the proposal was being developed (eg here https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-June/016057.html) 10:19 < glozow> afaiu, a guideline for web APIs 10:19 < larryruane> (not standard in the specific messages and their meaning, but the mechanism) 10:19 < stickies-v> A stateless representation of resources, typically (but I think not always?) communicated over HTTP with HTTP status codes. The stateless part basically means that the webserver has no state, i.e. each request is fully self contained and does not depend on e.g. previous requests. This makes for much more easily scalable services, since it doesn't matter which of many servers handles your request - there is noo state 10:19 < stickies-v> anyway. 10:20 < esraa> a request to the server over http/s 10:21 < tr3xx> REST is short for Representational State Transfer and is usually used to send data over HTTP/S 10:22 < dergoegge> from what i gathered it is a very common client-server architectural style for designing APIs that are scalable and simple. 10:22 < dergoegge> pretty much what y'all just said 10:23 < dergoegge> stickies-v: the "stateless" part is always mentioned when talking about REST so that seems like a core concept here. 10:24 < stickies-v> yeah, unfortunately that's both the most important and the most difficult to grasp part :( 10:24 < larryruane> ".. stateless.." this means the server doesn't need to maintain any per-client state (and time out this state, etc.) 10:25 < Kaizen_K_> ah yea, you dont need an individual session per client request 10:25 < Kaizen_K_> everyone making a REST call is equal 10:26 < stickies-v> larryruane the "per-client" qualification is very helpful to make it more understandable indeed, good point 10:26 < dergoegge> larryruane: +1 10:27 < dergoegge> next question: What are the main differences between the JSON-RPC and REST interface? 10:27 < Kaizen_K_> I understand the rest interface as being something public facing, just generally open to the publci. 10:27 < Kaizen_K_> *public. 10:28 < Kaizen_K_> The json-rpc would be something that the node operator would use when building a service ect 10:28 < Kaizen_K_> like having your lightning network node access your btc node would be done through rpc 10:28 < stickies-v> The REST api only offers a subset of functionality of the full fledged JSON-RPC. It is unauthenticated (but still weirdly somewhat trusted, so don't go ahead and open it up to the internet) and meant to expose public data in an easy and fast way. 10:28 < larryruane> One big difference probably is that REST (the way bitcoind uses it) is read-only (query various stuff), while RPC can change things (like submit transactions) 10:29 < glozow> I have a dumb question, I've never built a web app. I thought REST was just a concept, not itself a communication protocol or tool. if someone says "REST API" is that shorthand for "our web API which is RESTful" or? 10:29 < Kaizen_K_> larryruane: that is insightfull, rest should be for read only, json-rpc is full command control 10:29 < larryruane> glozow: that's a great question! 10:29 < stickies-v> glozow indeed, REST is just a concept. It says nothing about the comms protocol, but HTTP(S) is by far the most used 10:29 < dergoegge> Kaizen_K_: it is not, neither the REST nor the JSON-RPC interface are recommended to be open to the public 10:30 < Kaizen_K_> dergoegge: good to know 10:30 < glozow> stickies-v: okie thank u i was a lil confused 10:30 < larryruane> I haven't checked but I assume the REST interface can't be used to extract secret material (such as keys)? 10:30 < michaelfolkson> I think the distinction between REST vs JSON-RPC generally and Core's REST vs Core's JSON-RPC could avoid confusion 10:30 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has joined #bitcoin-core-pr-reviews 10:31 < michaelfolkson> Some things are specific to Core 10:31 < dergoegge> the REST interface can serve the contents of your mempool for example, so don't open it to the public! 10:31 < sipa> stickies-v: it's "trusted" in the sense that if you expose it to the internet, you risk being DoS'ed 10:31 < sipa> it has no privileged access 10:31 < larryruane> dergoegge: can you elaborate why exposing the content of your mempool is bad? 10:32 < stickies-v> sipa ah that's good to know - and relevant to the last question of this PR review I suppose, thanks! 10:32 < sipa> well, and privacy, i guess 10:32 < dergoegge> larryruane: exposing the content of your mempool is v bad for privacy, an attacker could figure out which transactions are broadcast by you 10:32 < larryruane> dergoegge: +1 10:33 < dergoegge> so one big difference is that JSON-RPC is authenticated while the REST interface is not 10:33 < Kaizen_K_> dergoegge: ty, that makes sense, a hostile party could filter peoples transactions to deny them entry into a block. 10:34 < jnewbery> I also think that it's just generally not designed to be exposed on a public network. If you wanted to serve REST clients on a public network, you should put the bitcoind server behind a proxy. 10:34 < dergoegge> jnewbery: +1 10:35 < Kaizen_K_> So why are there these two interfaces, that seem like they shouldn't even be exposed publicly, are available to use instead of just one? Is it a performance issue? 10:35 < sipa> michaelfolkson: obviously; you can't compare REST and JSON-RPC as concepts on their own, they're not comparable (one is a principle for client/server communication, the other is a specific protocol) 10:35 < dergoegge> Kaizen_K_: as long as one miner is honest this won't happen 10:35 < Kaizen_K_> dergoegg: ty 10:35 < sipa> Kaizen_K_: REST is way more convenient to use 10:35 < jnewbery> Kaiken_K: that's question 5 :) 10:35 < dergoegge> Kaizen_K_: good question, this is also what the last question today will be about 10:36 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has quit [Ping timeout: 245 seconds] 10:36 < sipa> < larryruane> I haven't checked but I assume the REST interface can't be used to extract secret material (such as keys)? <-- absolutely; only public data is available through REST 10:36 < Kaizen_K_> So from what I gather, it must be a qol issue for developers. 10:37 < Kaizen_K_> somethings are just too much a paid in the ass to authenticate for 10:37 < dergoegge> just in short my understing of the two interfaces: The JSON-RPC interface is used to control your node through the “bitcoin-cli” binary while the REST interface is there to serve public data (blocks, txs, etc) to a trusted caller. 10:37 < jnewbery> sipa: public-ish. The contents of your mempool are semi-public 10:37 < sipa> jnewbery: well, it's not *secret* data; it may be private (in the sense of privacy) 10:38 < sipa> but fair, that's a distinction 10:38 < jnewbery> sipa: agree 10:38 < jnewbery> dergoegge: bitcoin-cli is by far the most common client for the json-rpc interface, but it's perfectly possible to use other clients 10:39 < Kaizen_K_> :/s/paid/pain/g/ 10:39 < dergoegge> jnewbery: true, i forgot :D 10:39 < sipa> i'd be surprised if the number of RPC calls made with bitcoin-cli to bitcoind is even a measurable fraction of the total 10:39 < sipa> it's the most visible for sure, as it's half-way intended for human/developer interaction 10:40 < jnewbery> or maybe i'd restate as "it's perfectly possible for other applications to access that interface as a client" 10:40 < sipa> but anything automated will just use a JSON-RPC library 10:40 < dergoegge> lightning nodes also make significant use of the json-rpc interface 10:40 < larryruane> sipa: like our python tests! 10:40 < sipa> (except c-lightning, i think, which i don't comprehend...) 10:41 < dergoegge> next question: The JSON-RPC interface is already capable of serving blockfilters, why do we want this ability for the REST interface? 10:41 < Kaizen_K_> privacy? 10:41 < larryruane> dergoegge: my impression is that every lightning full node needs (or typically has) its own dedicated local bitcoind 10:41 < jnewbery> sipa: right. I retract my statement about bitcoin-cli being the most common client. It's only the most commonly used client amongst Bitcoin Core developers. 10:41 < sipa> jnewbery: agree 10:41 < dergoegge> larryruane: LND also an option to use their "neutrino" BIP158 light client 10:41 < michaelfolkson> c-lightning doesn't use a JSON-RPC library, it uses bitcoin-cli and this surprises you? Have I understood that right sipa? 10:42 < sipa> michaelfolkson: yes 10:42 < sipa> (or this used to be the case, at least; my information may be outdated) 10:42 < dergoegge> larryruane: although i think it is discouraged (not sure though) 10:42 < michaelfolkson> Why shouldn't it use bitcoin-cli? More restrictive? 10:42 < sipa> michaelfolkson: starting an entire new process for every RPC call? 10:42 < sipa> that's some ridiculous overhead 10:42 < stickies-v> dergoegge REST apis are super easy to consume, especially with all the tooling built around it (e.g. OpenAPI, although I've not seen an OpenAPI spec for bitcoin yet). I think developer ease of use is the main reason? 10:43 < Kaizen_K_> Um, I think we are veering off topic, I don't really mind, I'm still learning anyway. 10:43 < michaelfolkson> Sorry, just found that very interesting :) 10:43 < Kaizen_K_> stickies-v: thats my intuition as well, quality of life and privacy 10:43 < Kaizen_K_> michaelfolkson: np :) 10:44 < dergoegge> stickies-v: yes ease of use and lack of authentication make the REST interface attractive in certain use cases 10:44 < larryruane> dergoegge: "why do we want this ability for REST" -- it would have been nice if the PR answered this, I'm wondering too 10:45 < dergoegge> i also think it just makes sense to also serve the blockfilters as they are also public data like blocks or transactions 10:45 < larryruane> (unless it's just sort of a desire for more completeness) 10:45 -!- varioust [~varioust@72-46-48-28.lnk.ne.static.allophone.net] has quit [Ping timeout: 260 seconds] 10:45 < stickies-v> does anyone know if there has been any discussion around making an OpenAPI spec for the REST API? If not, I could look into contributing that 10:45 < jnewbery> The REST interface should also be more performant (at least in theory), since it can return binary data. The json-rpc interface always serializes its returned data into json text. If your application is going to immediately deserialize that, then it's unnecessary overhead. 10:45 < dergoegge> larryruane: yeah matt did not mention any use cases 10:45 < Kaizen_K_> larryruane: thats what I'm wondering too, I wonder if this is a scalability issue, RPC creates more network traffic through authentication? Rest maybe reduces that? 10:46 < sipa> REST interface is binary (or at least can be) 10:46 < dergoegge> jnewbery: good point the REST interface supports different formats 10:46 < sipa> JSON-RPC needs hex + JSON encoding/decoding on both sides 10:46 < sipa> oh, i'm repeating what jnewbery said 10:46 -!- svav [~svav@82-69-86-143.dsl.in-addr.zen.co.uk] has quit [Quit: Connection closed] 10:46 < Kaizen_K_> Ah interesting 10:46 < jnewbery> sipa: that's ok. The validation is nice :) 10:46 < larryruane> just to complete the picture, we have these two, plus there's ZMQ 10:47 < sipa> Kaizen_K_: it's not the authentication that's the issue; it's just that pumping large binary blobs through JSON is kind of dumb 10:47 < Kaizen_K_> I see I see 10:47 < Kaizen_K_> Rest gives a nice performance optimization on encoding/decoding and maybe less network traffic, so its all around more performant 10:48 < sipa> and just... easier 10:48 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has joined #bitcoin-core-pr-reviews 10:48 < Kaizen_K_> + the qol 10:48 < Kaizen_K_> cool cool 10:48 < dergoegge> sipa: +1 10:48 < Kaizen_K_> shit man, these pr-clubs are super informative 10:48 < jnewbery> larryruane: that's right. ZMQ is the third interface, but it's a completely different animal 10:49 < larryruane> my impression of ZMQ is that it allows notifications, so the client doesn't have to continuously poll 10:49 < Kaizen_K_> I've always wondered what amq was, I know its important with lightning-d 10:49 < Kaizen_K_> lnd sry 10:50 < Kaizen_K_> TIL: ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the 10:50 < Kaizen_K_> fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. 10:50 < larryruane> Kaizen_K_: "are super informative" Yes, and be aware that you can go back and read all the old ones! (which I haven't done enough myself!) 10:50 < Kaizen_K_> larryruane: thats a good suggestion, I'm going to do that 10:51 < dergoegge> Kaizen_K_ all meeting logs can be found here: https://bitcoincore.reviews/meetings/ 10:51 < Kaizen_K_> This zeromq sounds similar to google-protobuffers or am I mistaken? 10:51 < michaelfolkson> https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md 10:51 < dergoegge> ok last question: There is a NACK (#23259) on the PR suggesting that the REST interface should be removed entirely in favour of external proxy servers. Do you agree? Why/why not? 10:51 < dergoegge> https://github.com/bitcoin/bitcoin/issues/23259 10:52 < Kaizen_K_> After this meeting, I disagree, a fair amount of data can be optimized 10:52 < Kaizen_K_> Rest is good 10:52 < dergoegge> Jeremy also made a PR to exemplify an external proxy server on top of the JSON-RPC interface: https://github.com/bitcoin/bitcoin/pull/23309 10:54 < jnewbery> dergoegge: thanks for pointing that out. I wasn't aware of that PR 10:54 < stickies-v> dergoegge I mostly agree. If we're trying to keep core reviewable and maintainable, I see this as an easy to carve out some code - even though it is tiny. Like Jeremy proposes, I think it would make sense to have a separate project to run a full fledged REST server (ideally with optional auth to access the full feature set that JSON RPC represents). However, I don't think the REST functionality is a particular 10:54 < stickies-v> attack vector since it's so simple, so that's why I only mostly agree. 10:54 -!- Kaizen_Kintsugi [Kaizen_Kin@gateway/vpn/protonvpn/kaizenkintsugi/x-99714370] has quit [Ping timeout: 264 seconds] 10:54 < stickies-v> *mostly agree with Jeremy, that is 10:55 < jnewbery> I disagree that we should remove the REST interface. I expect that a lot of people are using it. 10:56 < michaelfolkson> Other than additional maintenance why not just add an external proxy server as an additional option rather than as a replacement for REST? 10:56 < sipa> what does "just add an additional proxy server" mean? 10:56 < dergoegge> stickies-v should that stop this PR from getting merged though? removing the REST interface is difficult if it has actual users 10:56 < jnewbery> michaelfolkson: because it's not the responsibility of the bitcoin core developers/maintainers to write/test/maintain a proxy server 10:57 < dergoegge> jnewbery: +1 10:57 < stickies-v> stickies-v no I don't think it should stop this PR, this is a useful addition. I think they're separate. This PR is about getting the functionality up to date, Jeremy's PR is about carving out that functionality into a different project I think? 10:57 < jnewbery> the responsibility of the Bitcoin Core project ends at the interface. How those interfaces are consumed/used is the responsibility of the client user/application 10:57 < sipa> i think there is little point in removing the REST server; it's simple, straightforward code with barely a maintenance burden 10:58 < sipa> and removing it makes the functionality it provides harder to use (separate server etc) 10:58 < dergoegge> stickies-v he is proposing the proxy to still be in the core repo just not part of the binary 10:59 < michaelfolkson> I think Jeremy's argument is that people shouldn't be using REST. e.g. the discussion on sanitizing 10:59 < michaelfolkson> "exposing this rest endpoint over NGINX is precisely how our rest endpoint should not be used" 10:59 < dergoegge> sipa: i agree 10:59 -!- Azorcode [~Azorcode@201.242.162.105] has quit [Ping timeout: 260 seconds] 10:59 < michaelfolkson> Kinda protecting the user (don't necessarily agree but I think that is the argument for removal) 11:00 -!- raj [~raj@2602:ffb6:4:e396:f816:3eff:fe47:ca2a] has quit [Killed (NickServ (GHOST command used by raj_!uid72176@user/raj))] 11:00 < sipa> i think the REST interface just shouldn't be exposed to the internet 11:00 < dergoegge> #endmeeting 11:00 -!- raj [~raj@167.182.81.172.lunanode-rdns.com] has joined #bitcoin-core-pr-reviews 11:00 < stickies-v> michaelfolkson I'm not sure that's what he's after, in https://github.com/bitcoin/bitcoin/issues/23259#issuecomment-940648658 he mentions that he's open to replacing the json rpc with rest entirely? 11:00 < sipa> with or without proxy 11:00 < dergoegge> thanks for coming everyone! 11:00 < dergoegge> feel free to stay and discuss 11:00 -!- raj [~raj@167.182.81.172.lunanode-rdns.com] has quit [Killed (NickServ (GHOST command used by raj_!uid72176@user/raj))] 11:00 < glozow> thanks dergoegge! 11:00 < Kaizen_K_> dergoegge: thanks for hosting, I learned a lot 11:00 < sipa> if you really want to expose it, yes, use a wrapper to sanitize it 11:01 < sipa> but that's not its intended use case 11:01 -!- raj_ [~raj@167.182.81.172.lunanode-rdns.com] has joined #bitcoin-core-pr-reviews 11:01 < stickies-v> thanks for hosting dergoegge , very vibrant meeting! (and thanks Matt for the PR) 11:01 < larryruane> PR 23309 says `[WIP]` (work in progress) instead of making the PR a draft -- is the GitHub draft PR feature not used much in Core? 11:01 < larryruane> thanks, dergoegge this was great!! 11:01 < sipa> #23309 11:01 < Kaizen_K_> when does the next PR go up? 11:01 < Kaizen_K_> I wanna be more prepared for next week 11:01 < sipa> what "next PR" ? 11:01 < Kaizen_K_> I guess the next PR review 11:01 < sipa> ah, next review club? 11:02 < jnewbery> dergoegge: great job. Thank you! 11:02 < glozow> #22674 next week :) 11:02 < larryruane> Kaizen_K_: around Friday usually 11:02 < Kaizen_K_> cool cool 11:02 < glozow> https://github.com/bitcoin/bitcoin/pull/22674 11:02 < michaelfolkson> Yeah really interesting, thank you dergoegge 11:02 < sipa> larryruane: the author may be unfamiliar with the feature 11:02 < dergoegge> Kaizen_K_ if you lurk on this repo: https://github.com/bitcoin-core-review-club/website you will always be up to date on the next meeting 11:03 < michaelfolkson> Yeah you can watch it and get email notifications 11:03 < michaelfolkson> Beware you get a lot of emails if you watch a few repos :) 11:03 < larryruane> after these meetings, I always have a ton of new broswer tabs open that I need to read, haha! 11:03 < larryruane> michaelfolkson: thanks, TIL! (notifications) 11:04 < Kaizen_K_> yea me too, it's really opening me up to my knowledge gaps 11:04 < jnewbery> Also follow https://twitter.com/BitcoinCorePRs for all updates 11:04 < stickies-v> I'm not sure if anyone has any thoughts on this, but my Approach NACK is because I don't think should be a path parameter in `GET /rest/blockfilterheaders///.` - it's not restful. 11:04 < stickies-v> Instead, I think this should be a query parameter, for example `GET /rest/blockfilterheaders//.?count=`. Thoughts? 11:04 < jnewbery> and there's also a feed on the website: https://bitcoincore.reviews/feed.xml 11:04 < larryruane> jnewbery: I do follow but darn twitter doesn't show me its tweets! 11:04 < stickies-v> (and also - does a reservation like that warrant an Approach NACK?) 11:06 < tr3xx> This was great, I have multiple tabs open as well! It was fun lurking, watching the discussions :) 11:06 < larryruane> BTW in case anyone hasn't tried fetching a block filter yet, I ran `bitcoin-cli getblockfilter 00000000000000000006f9a460e2f86f4262d8970902f7f38b0f86bf08bfc898` and got the error `Index is not enabled for filtertype basic` 11:06 < michaelfolkson> stickies-v: I think suggest it as a change and then if the author doesn't want to make the change it is up to you whether you would rather the PR wasn't merged because of it (whether it is worthy of an Approach NACK) 11:06 < sipa> stickies-v: voicing the actual objection would certainly be far more productive than a blanket nack 11:06 < michaelfolkson> You can always open an alternative PR if you feel that strongly about it 11:07 < dergoegge> larryruane: did you run your node with the `-blockfilterindex` option? 11:07 < larryruane> so i added `blockfilterindex=1` to my config file, restarted, and now it's building these filters, starting from block 0, in the background (still adding new blocks) 11:07 < jnewbery> stickies-v: you should definitely raise specific objections in review. I find that the word NACK tends to antagonize people, so I use it sparingly and only when I think the PR is harmful to the project. 11:08 < stickies-v> michaelfolkson sipa jnewbery got it, thanks for the advice on how to approach this- that makes sense! Will start with the suggestion first. 11:08 < sipa> stickies-v: i'd reserve an approach nack for "you're doing this completely the wrong, the whole thing needs to be done differently" 11:08 < larryruane> dergoegge: yes .. but it's going very slowly, it's up to height 1537 after about an hour 11:09 < larryruane> at this rate it will take weeks or months ... but then it will be a service to the network! 11:09 < dergoegge> larryruane: oof, thats seems way too slow 11:09 < dergoegge> what hardware is this on? :D 11:09 < Kaizen_K_> I just moved and I'm on potato internet, not looking forward to dling the blockchain 11:10 < larryruane> dergoegge: the HW is probably the problem, it's running on my fast laptop, but the disk is USB non-SSD 11:10 < larryruane> it took several weeks to sync (IBD) with this disk 11:11 < dergoegge> larryruane: ah i see 11:11 < larryruane> dergoegge: did you enable that option? How long did it take to sync the filters? 11:11 -!- Naiza [~Naiza@180.188.251.235] has quit [Quit: Connection closed] 11:12 < dergoegge> never had it enabled on mainnet, but on signet it took a couple minutes max 11:12 < larryruane> oh ok, yes, this is mainnet 11:13 < dergoegge> oh wait no i did it on mainnet for a review recently, left it running over night and the next morning it was finished 11:13 < dergoegge> nvme ssd though so... 11:13 < larryruane> ok.. it's nice that it does it in the background 11:14 < sipa> larryruane: whatever you do, Do. Not. Put. The. Datadir. On. A. USB. Drive. 11:14 -!- ajayparmar904 [~ajayparma@103.39.129.252] has quit [Quit: Connection closed] 11:15 < larryruane> OH, really? why? performance? 11:15 < sipa> consumer usb controllers aren't designed for that kind of load 11:15 < sipa> it'll corrupt your data 11:15 < larryruane> i see! okay, thanks!! 11:15 < stickies-v> michaelfolkson sipa jnewbery followed your advice on https://github.com/bitcoin/bitcoin/pull/17631#discussion_r733031180 - thanks again! 11:22 -!- tr3xx [~tr3xx@2a01:799:5f:ec00:d9b9:6729:9569:c6ea] has quit [Quit: Leaving] 11:24 -!- stickies-v [~stickies-@84-45-243-170.static.enta.net] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 11:27 -!- varioust [~varioust@72-46-48-28.lnk.ne.static.allophone.net] has joined #bitcoin-core-pr-reviews 11:28 -!- esraa [~esraa@147.236.159.129] has quit [Ping timeout: 260 seconds] 11:55 -!- yonson [~yonson@2600:8801:d900:7bb:1e69:7aff:fea2:4e85] has quit [Remote host closed the connection] 11:56 -!- yonson [~yonson@2600:8801:d900:7bb:1e69:7aff:fea2:4e85] has joined #bitcoin-core-pr-reviews 12:05 -!- Talkless [~Talkless@mail.dargis.net] has quit [Quit: Konversation terminated!] 12:37 -!- varioust [~varioust@72-46-48-28.lnk.ne.static.allophone.net] has quit [Ping timeout: 260 seconds] 12:45 -!- Kaizen_K_ [~Kaizen_Ki@node-1w7jr9yi65te55mxt1kkm1n0k.ipv6.telus.net] has quit [Remote host closed the connection] 12:45 -!- Kaizen_Kintsugi [~Kaizen_Ki@node-1w7jr9yi65te55mxt1kkm1n0k.ipv6.telus.net] has joined #bitcoin-core-pr-reviews 12:50 -!- Kaizen_Kintsugi [~Kaizen_Ki@node-1w7jr9yi65te55mxt1kkm1n0k.ipv6.telus.net] has quit [Ping timeout: 252 seconds] 13:26 -!- Kaizen_Kintsugi [~Kaizen_Ki@node-1w7jr9yi65te55mxt1kkm1n0k.ipv6.telus.net] has joined #bitcoin-core-pr-reviews 13:30 -!- Kaizen_Kintsugi [~Kaizen_Ki@node-1w7jr9yi65te55mxt1kkm1n0k.ipv6.telus.net] has quit [Remote host closed the connection] 13:30 -!- Kaizen_Kintsugi [~Kaizen_Ki@node-1w7jr9yi65te55mxt1kkm1n0k.ipv6.telus.net] has joined #bitcoin-core-pr-reviews 14:40 -!- varioust [varioust@cpe-108-167-11-88.neb.res.rr.com] has joined #bitcoin-core-pr-reviews 14:47 -!- varioust [varioust@cpe-108-167-11-88.neb.res.rr.com] has quit [Ping timeout: 258 seconds] 14:47 -!- varioust [~varioust@cpe-108-167-11-88.neb.res.rr.com] has joined #bitcoin-core-pr-reviews 15:29 -!- dunxen [~dunxen@gateway/tor-sasl/dunxen] has quit [Ping timeout: 276 seconds] 16:27 -!- varioust [~varioust@cpe-108-167-11-88.neb.res.rr.com] has quit [Ping timeout: 258 seconds] 16:49 -!- varioust [~varioust@104-218-67-9.dynamic.allophone.net] has joined #bitcoin-core-pr-reviews 16:50 -!- varioust [~varioust@104-218-67-9.dynamic.allophone.net] has quit [Remote host closed the connection] 18:06 -!- pg2 [sid518209@id-518209.hampstead.irccloud.com] has quit [Ping timeout: 264 seconds] 18:07 -!- schmidty [sid297174@id-297174.lymington.irccloud.com] has quit [Ping timeout: 264 seconds] 18:07 -!- jamesob [uid180710@id-180710.helmsley.irccloud.com] has quit [Ping timeout: 252 seconds] 18:12 -!- jamesob [uid180710@id-180710.helmsley.irccloud.com] has joined #bitcoin-core-pr-reviews 18:12 -!- stick [sid403625@user/prusnak] has quit [Ping timeout: 252 seconds] 18:15 -!- stick [sid403625@user/prusnak] has joined #bitcoin-core-pr-reviews 18:21 -!- pg2 [sid518209@id-518209.hampstead.irccloud.com] has joined #bitcoin-core-pr-reviews 18:21 -!- schmidty [sid297174@id-297174.lymington.irccloud.com] has joined #bitcoin-core-pr-reviews 21:20 -!- belcher [~belcher@user/belcher] has quit [Ping timeout: 260 seconds] 21:34 -!- belcher [~belcher@user/belcher] has joined #bitcoin-core-pr-reviews 23:07 -!- TheRec [~toto@user/therec] has quit [Read error: Connection reset by peer] 23:07 -!- TheRec [~toto@84-75-225-47.dclient.hispeed.ch] has joined #bitcoin-core-pr-reviews 23:07 -!- TheRec [~toto@84-75-225-47.dclient.hispeed.ch] has quit [Changing host] 23:07 -!- TheRec [~toto@user/therec] has joined #bitcoin-core-pr-reviews --- Log closed Thu Oct 21 00:00:10 2021