--- Log opened Mon Jul 18 00:00:24 2022
01:37 -!- meshcollider [meshcollid@meshcollider.jujube.ircnow.org] has joined #secp256k1
06:50 < real_or_random> reminder that our biweekly dev meeting takes place today at 15:00 UTC (in 70 min)
06:52 -!- halosghost [~halosghos@user/halosghost] has joined #secp256k1
07:00 -!- ghost43 [~ghost43@gateway/tor-sasl/ghost43] has quit [Remote host closed the connection]
07:00 -!- ghost43 [~ghost43@gateway/tor-sasl/ghost43] has joined #secp256k1
08:01 < real_or_random> meeting!
08:01 < hebasto> hi
08:01 < real_or_random> hi
08:01 < jesseposner> hi
08:01 < sipa> hi
08:01 < fanquake> hi
08:01 < siv2r[m]> hi
08:02 < real_or_random> nickler 
08:03 < nickler> sup
08:03 < real_or_random> :)
08:03 < sipa> sky
08:04 < real_or_random> i can't follow
08:04 < real_or_random> topics? 
08:04 < sipa> nickler asked what was up. the sky is up
08:04 < real_or_random> -.-
08:04 < nickler> if siv2r wants he can give a status update of his batch verify project
08:05 < real_or_random> :)
08:05 < sipa> oh cool
08:05 < siv2r[m]> sure
08:06 < nickler> also, does #1126 fully address the concerns that are blocking an initial release?
08:06 < siv2r[m]> For those who don't know me, I have been working on a batch verification interface for libsecp as part of summer of bitcoin.
08:06 < siv2r[m]> I have implemented an experimental batch module (APIs, tests, and benchmarks). 
08:06 < siv2r[m]> Link: https://github.com/siv2r/secp256k1/pull/2
08:06 < siv2r[m]> The PR is almost ready for review.
08:07 < siv2r[m]> There are some TODOs left: 
08:07 < siv2r[m]> (1) tweak the APIs for better batch verification speed-up
08:07 < siv2r[m]> (2) rewrite commit history.
08:07 < siv2r[m]> If things go smoothly, I will send the PR by this week.
08:08 < real_or_random> nice! does this include improvements to the ecmult batch / scratch code?
08:08 < real_or_random> or is purely built on top?
08:09 < siv2r[m]> I have refactored the `emcult_strauss_batch` to support scratch space pre-filled with scalars and points
08:10 < real_or_random> if we're talking about this, we can also talk about our confidence in the ecmult multi code
08:11 < siv2r[m]> siv2r[m]: Also, the for computing multi scalar multiplication only strauss is used, it doesn't switch to pippegger (after threshold) like `ecmult_multi_var`
08:11 < real_or_random> I see
08:12 < sipa> Is that something you plan on addresses? The gains for batches are bigger when you get into pippenger territory.
08:13 < nickler> fwiw using strauss with a large number of points quickly gets slower (at <500 sigs iirc) than single verification
08:14 < nickler> siv2r's code integrates pretty easily with the strauss api. It'd be quite a bit more complex if we also supported pippenger.
08:14 < real_or_random> it gets slower than single verification?
08:15 < nickler> So just using strauss for now is reasonable to me. I'm generally not too confident in the ecmult_multi code, so it's good to reduce complexity.
08:16 < nickler> real_or_random: yes
08:16 < siv2r[m]> Yes. I haven't included pippenger since, it required a lot of changes to the batch object. The major problem was pippenge represents points as `secp256k1_ge` whereas `strauss_wnaf` represents points as `secp256k1_gej` (in scratch). So, I was not able to use the same scratch.
08:16 < nickler> there's some optimum at 50 to 150 sigs
08:17 < real_or_random> oh that makes sense
08:17 < sipa> https://bitcoin.sipa.be/speedup-aggsig.png
08:17 < sipa> Old graph, but that does suggest differently.
08:17 < real_or_random> hm ok, the interface is only superficially the same...
08:18 < sipa> This one is a bit more relevant (but equally outdated): https://bitcoin.sipa.be/speedup-batch.png
08:18 < nickler> sipa: interesting, there could also be a bug in the code
08:19 < sipa> Just to confirm... the idea is to only support strauss initially, but the (public) batch validation API wouldn't change if we'd want to make it support pippenger too?
08:20 < nickler> that's the best idea I know about so far :)
08:20 < real_or_random> could be something related to memory management. maybe at some point that becomes too expensive due to a performance bug
08:21 < real_or_random> regarding confidence: all the ecmult multi code has been lying around as dead code (except tests) for a while and I think it had a few bugs and code smells when it comes to manual memory management (scratch)
08:22 < real_or_random> so do you think it would make sense to re-review it?
08:23 < real_or_random> sipa: how did you obtain the graphs? did the code actually perform verification of signatures held in memory or was this just ecmult multi?
08:23 < nickler> the strauss code isn't dead, right? we've also had a look at it a few months ago when roconnor did some code improvements (that I don't remember)
08:23 < sipa> PR 1044 did add additional tests for it too
08:23 < real_or_random> I mean dead in the sense that it's not used in the public API 
08:24 < nickler> but in the ecmult_multi code outside of strauss there are definitely a bunch of integer overflows for example
08:25 < real_or_random> we should definitively restrict the number of signatures in batch
08:26 < real_or_random> *in one batch
08:26 < sipa> yeah, reasoning about the memory usage and scratches is certainly something that i'm more worried about than the multiplication logic itself
08:27 < nickler> another idea is that instead of allowing every possible scratch space size for batch verification, just give options for "small" "medium" "large". This is easier to test.
08:27 < sipa> real_or_random: I do not remember how those graphs were created.
08:27 < real_or_random> sipa: :D
08:29 < nickler> btw, in the graph if you go beyond 10,000 sigs one can imagine that the line drops lower than the y=1.4 that the strauss-endo line has at x=0
08:29 < nickler> so perhaps there's no bug in siv's code
08:30 < nickler> running his code, at 10000 sigs it's about as fast as individual verification. It's definitely slower at 20000
08:30 < sipa> The changes in times for inversions may have pretty radically changes the profile.
08:31 < real_or_random> ok, and 10000 another number than 500
08:31 < real_or_random> is
08:31 < real_or_random> (in terms of practical relevance)
08:31 < nickler> sorry, the 500 number was for a different version of siv's code
08:31 < sipa> Oh, when benchmarking batch schnorrsig validation, a realistic measurement should also include the time to deserialize the pubkeys (which involves a square root for each).
08:32 < sipa> I think my graphs don't do that.
08:32 < nickler> we did a bunch of optimizations today (mainly restricting and shifting the randomizer range)
08:32 < real_or_random> yep, that's for example what I was asking today
08:32 < nickler> siv's benchmarks do that
08:32 < sipa> Cool.
08:32 < real_or_random> anyway, exiting to see a PR
08:33 < nickler> bye real_or_random ;P
08:33 < sipa> Indeed.
08:33 < siv2r[m]> :)
08:33 < sipa> @nickler :D
08:33 < real_or_random> ...
08:33 < real_or_random> oops
08:34 < real_or_random> next topic? :D
08:35 < nickler> k, does #1126 fully address the concerns that are blocking an initial release?
08:35 < real_or_random> at least my concerns, yes
08:36 < nickler> good!
08:37 < nickler> I don't remember any other blockers, sipa?
08:37 < sipa> Neither do I.
08:38 < nickler> next topic?
08:38 < real_or_random> that was quick, yes...
08:38 < sipa> I said I'd go through the release PR again; I forgot I was OOO past week though; I'll do so in the next days.
08:38 < nickler> cool
08:39 < nickler> I also promised to make progress with synthetic int, but I didn't really
08:39 < sipa> Are there cmake updates?
08:39 < hebasto> all current comments have been addressed
08:40 < sipa> Cool, will have another look soon.
08:40 < hebasto> thanks!
08:40 < real_or_random> I think the question for me is if we should we rework the config system first 
08:40 < sipa> As in moving further along the "no external config needed" linr?
08:40 < real_or_random> though now that I think about it, that shouldn't block merging the cmake PR
08:41 < real_or_random> yes, this is related is when it comes to docs 
08:41 < real_or_random> ahh hm now I remember. So what I don't like about the cmake PR is that it adds another config file
08:42 < sipa> Oh, yes.
08:42 < real_or_random> so then we have config options document in autoconf help and in cmake. and not for manual users 
08:43 < real_or_random> what I have in mind is to move almost all config docs to the README or some other .md
08:43 < hebasto> like `build.md`?
08:44 < real_or_random> then the build system would at most have very generic descriptions such as "enable x module" without going into details, and we would have all the docs in a single place
08:44 < real_or_random> yep
08:44 < sipa> i like that
08:44 < real_or_random> (though our readme is still short, so it could also go there)
08:45 < hebasto> could doc improvements go as a follow up PR?
08:46 < real_or_random> I guess ideally this would be a separate PR that comes first. Though I don't think it's strictly necessary
08:46 < sipa> The config improvement is a bit more fundamental I think.
08:47 < hebasto> right
08:47 < real_or_random> if you'd like to get the PR off your plate, we'd just need to see what to do with the config for now
08:49 < real_or_random> hebasto: what do you think?
08:50 < hebasto> you mean about a separate docs PR which should go first?
08:50 < sipa> The config improvements is not the same as the documentation.
08:51 < sipa> I think?
08:51 < real_or_random> I meant do you think we should wait until the config is cleaned up or would you prefer to get the cmake PR quickly off your plate?
08:51 < real_or_random> sipa: ookkk, yes, I'd say these are two parts of the same projec 
08:52 < hebasto> I'm ok with cleaning up config first
08:52 < sipa> Sure, but the config part is something we'd want before cmake I think - docs improvements can happen anytime.
08:52 < hebasto> I will still maintain the PR in the mergable state
08:52 < real_or_random> I think the doc part is the only one interfering with the cmake PR : if we have the docs,  then the cmake PR won't need to include a config file  
08:53 < hebasto> hmm, I don't think so
08:53 < real_or_random> sipa: ok :D my point would have been that the config part is mostly renaming macros. and this is something we could always do 
08:54 < real_or_random> anyway, let's just do both parts before cmake then
08:54 < hebasto> agree
08:54 < real_or_random> hebasto: so you think we'd still have a config file for cmake?
08:54 < sipa> sgtm
08:55 < real_or_random> by the way, talking about the README, ... should we mention the IRC channel (and maybe the meeting here)?
08:55 < sipa> Not sure.
08:55 < hebasto> if build, in general, will support config-less routine, cmake won't require it
08:56 < sipa> I could mention the github reposutory though
08:57 < real_or_random> sipa: the readme? for users that get tarballs?
08:57 < real_or_random> hebasto: ok, then we're on the same page
08:58 < sipa> Yeah, in the readme. It means that if someone finds a clone of the repo, or a subtree of it somewhere, they know how to locate upstream.
08:59 < real_or_random> makes sense
08:59 < real_or_random> and IRC? do you fear that we'll see some of the more annoying folks here? 
09:00 < sipa> I guess we can mention the IRC channel too.
09:00 < sipa> Both the README and the meetings are intended for people building applications with the library, I guess.
09:01 < real_or_random> sgtm
09:05 < real_or_random> ok, anything else? 
09:05 < sipa> End of meeting, I guess?
09:06 < real_or_random> yep I think so
11:33 -!- roconnor [~roconnor@coq/roconnor] has joined #secp256k1
15:35 -!- halosghost [~halosghos@user/halosghost] has quit [Quit: WeeChat 3.5]
17:00 -!- Netsplit *.net <-> *.split quits: waxwing_, kanzure
17:00 -!- waxwing [~waxwing@193.29.57.116] has joined #secp256k1
17:45 -!- kanzure [~kanzure@user/kanzure] has joined #secp256k1
20:46 -!- ghost43 [~ghost43@gateway/tor-sasl/ghost43] has quit [Read error: Connection reset by peer]
20:46 -!- ghost43 [~ghost43@gateway/tor-sasl/ghost43] has joined #secp256k1
--- Log closed Tue Jul 19 00:00:25 2022