--- Log opened Sat Jul 06 00:00:08 2019
00:08 < fanquake> dongcarl: Have you completed any builds inside the docker container? Have any hashes I can compare against?
05:47 < fanquake> dw, builds are matching now.
09:01 < dongcarl> fanquake: Heyo
09:01 < dongcarl> Yay traffic!
12:28 < fanquake> 🏎
12:36 -!- hebasto [~hebasto@95.164.65.194] has quit [Remote host closed the connection]
14:22 < nothingmuch> dongcarl: bikeshedding dilemma re depends package
14:22 < nothingmuch> well, not exactly bikeshedding, but almost
14:22 < dongcarl> ?
14:23 < nothingmuch> so basically to substitute make -C depends download, we can convert depends/packages/*.mk into guix origin objects and stick those in as inputs
14:23 < nothingmuch> one possible approach is to use print-% or make -qp to let make evaluate those and spit out corresponding scheme source
14:23 < nothingmuch> or even invoke that dynamically from scheme in the package definition
14:24 < nothingmuch> and the other is to read the .mk files directly and evaluate the variables using simple string substitution
14:24 < nothingmuch> (i don't like invoking dynamically in the package definition, that seems like a big environmental dependency, so brittle)
14:25 < dongcarl> nothingmuch: Do it manually first
14:25 < dongcarl> we can make an importer later
14:25 < nothingmuch> i did, it works for the first 2 packages
14:25 < dongcarl> What doesn't it work for?
14:25 < nothingmuch> oh, i have no reason to expect it won't
14:25 < nothingmuch> i'm just too lazy to do them all by hand
14:26 < nothingmuch> i figured if it has to be done anyway eventually, i might as well just do it
14:26 < dongcarl> nothingmuch: So the first 2 packages built and produced outputs?
14:26 < nothingmuch> no this is just the origin objects used as inputs
14:27 < nothingmuch> so that we can construct SOURCES_PATH without network access
14:27 < dongcarl> Right, I'm saying you should get it to a point where outputs are produced for a single depends package
14:27 < dongcarl> Then try to figure out the details
14:27 < dongcarl> Does that make sense?
14:28 < nothingmuch> i'm not sure, because that means changing the depends makefile
14:28 < nothingmuch> all i want is to provide the tarballs for vanilla make depends first, without altering the deps' actual build process
14:29 < dongcarl> why does it require changing the depends makefile?
14:29 < nothingmuch> in order to avoid trying to build deps that the top level ./configure should find as a guix input
14:30 < nothingmuch> it all has to happen eventually, so i don't mind spending time on that either
14:31 < nothingmuch> but i thought a less ambitious goal would be first to make a package that does all of make depends as its currently done, and use that in the guix environment
14:31 < dongcarl> I'm not sure I'm following 100%
14:31 < dongcarl> Could you tell me what the less ambitious goal is?
14:32 < nothingmuch> so right now the build.sh script does 3 things as i see it - make depends, make dist, and per target builds
14:32 < dongcarl> right
14:33 < nothingmuch> ideally this make depends should be a sort of meta package, and make dist as a package, which the build package depends on as input
14:33 < nothingmuch> and the depends meta package would in turn take as input a package for every dependency
14:34 < nothingmuch> so focusing only on the make depends part, there are two ways to go about refactoring things:
14:34 < nothingmuch> 1. start by extracting individual dependencies into separate packages, and include them in the manifest
14:35 < nothingmuch> 2. start by just splitting up the entire make depends phase, so that it's not a meta-package but a package that actually runs make depends
14:35 < nothingmuch> i was assuming (2) is simpler, because in order to pull (1) off i thought i'd have to alter make depends, to avoid building stuff which is passed in as an input
14:36 < dongcarl> sec, eating
14:36 < nothingmuch> my reasoning basically is that i want to catch myself breaking determinism as early as possible
14:37 < dongcarl> I agree that those are the two ways that make sense
14:37 < dongcarl> Let's think thru them a little
14:38 < dongcarl> For (1)
14:38 < dongcarl> What will be included in the manifest will be "primary" dependencies
14:38 < dongcarl> As in, the ones that bitcoin core touches directly
14:39 < dongcarl> not things like libexpat that core doesn't use, but core's depencies use
14:39 < dongcarl> does that make sense?
14:39 < nothingmuch> yes
14:39 < nothingmuch> fwiw it's pretty easy to get a non qt build working from a git checkout with guix environment
14:39 < nothingmuch> by adapting the guix bitcoin-core package
14:40 < dongcarl> That's great. I think (1) is what we should aim for. Especially because that would be a significant win over the scripts I have now
14:41 < dongcarl> Once again... Beware of Guix's interaction with RPATHs and over-linking
14:41 < dongcarl> make sure that check-symbols passes
14:41 < dongcarl> https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/symbol-check.py
14:42 < nothingmuch> these subtleties are kind of what i'm worried about, which is why i thought (2) makes more sense - if there's a single package for all deps that still produces exactly the same outputs as the scripts currently do, for CI that is alreayd a big win
14:42 < nothingmuch> and the past work to ensure determinism is reused
14:42 < nothingmuch> or atleast if it diverges, that is deliberate
14:44  * dongcarl looking up things
14:46 < nothingmuch> actually, just to be clear: re (1) are we in agreement that these would be new packages, potentially redundant with guix's ones, that match the depends/packages/$pkg.mk build process, right?
14:46 < dongcarl> nothingmuch: Yes
14:46 < dongcarl> 99% sure they will be redundant
14:46  * dongcarl still thinking
14:46 < nothingmuch> and is the goal of trying to match the current output hashes from the scripts worthwhile?
14:46 < dongcarl> nothingmuch: no
14:46 < dongcarl> Not worthwhile
14:46 < nothingmuch> ok, that is good to know because i was assuming it is
14:47 < dongcarl> If we're going to change it, we're going to change it all
14:47 < dongcarl> I retract the above statement as a general statement but it makes sense in this context
14:47 < dongcarl> sec
14:50 < nothingmuch> the statement about change? fwiw, the reason i thought it was worthwhile is that i don't understand the previous of work that went into building deps the way that they are built, and preserving reproducibility with your scripts ensures i haven't lost any of it
14:51 < nothingmuch> so i guess equivalent outputs up to differences i can explain justify is more in line with what i was actually thinking
14:51 < dongcarl> So I think the progression might go like this:
14:51 < dongcarl> 1. The (2) as you stated before, where `make depends` is a single monolithic package
14:51 < dongcarl> 2. We still use `make depends`, but we now split everything up into packages, and do `make -C depends <pkg>_built`, with the right dependency declaration so that it doesn't mess up
14:51 < dongcarl> 3. The (1) as you stated before
14:52 < dongcarl> There will be some work that goes into 1.+2. that might be discarded for 3.
14:52 < dongcarl> So it's up to you if you want to do the progression or just jump to 3.
14:53 < nothingmuch> i don't mind discarding work, i'd rather have an iterative approach, convexity and all that
14:53 < dongcarl> That's wonderful to hear
14:53 < dongcarl> Can `origin` take local directories?
14:53 < dongcarl> Because if it can... I'd say just prototype with that
14:53 < nothingmuch> afaict no... but it's pretty easy to convert things
14:54 < nothingmuch> it's just simple string interpolation to parse out the makefile variables, which are pretty consistent
14:54 < nothingmuch> + base16 -> nix-base32 conversion for the sha256 hashes
14:54 < dongcarl> Sure if that's easy, I think you can also just tar up a `SOURCES_PATH`, put it on a web server, and use it as origin
14:55 < dongcarl> I'm assuming you've seen the depends doc for SOURCES_PATH?
14:55 < nothingmuch> yep, i was going to reconstruct SOURCES_PATH by just copying files from the various tarball inputs
14:56 < dongcarl> Lovely
14:56 < dongcarl> Yeah that all sounds really good :-)
14:56 < nothingmuch> actually i might just do that for now, with a webserver on localhost
14:56 < nothingmuch> to avoid bikeshedding until later
14:56 < dongcarl> Yup :-)
14:57 < nothingmuch> and i can overengineer conversion of *.mk files into packages once i'm convinced it all works
14:57 < nothingmuch> re (3) - do you envision static definitions of guix packages under source control?
14:57 < nothingmuch> or the .mk files still being ground truth and some sort of guix import workflow?
14:57 < nothingmuch> i was assuming the latter fwiw
14:57 < dongcarl> "I had to write all the code before I could convince myself that I could solve every problem"
14:57 < dongcarl> :-)
14:58 < nothingmuch> well, tbh, i found the orig bitcoin source pretty messy ;-)
14:58 < dongcarl> nothingmuch: I think we can decide when it comes to it...
14:58 < dongcarl> `guix import` would be very nice...
14:58 < nothingmuch> keeping guix package in source control would imply a separate repo too
14:59 < dongcarl> nothingmuch: Oh?
14:59 < dongcarl> I think we can just do what I do now
14:59 < dongcarl> And just keep it under `contrib/guix/packages`
14:59 < dongcarl> But anyways, I'd really like a guix import workflow
14:59 < nothingmuch> but then those packages can't refer to commits in the repo in which they are hosted
14:59 < dongcarl> Keep the source of truth the same
15:00 < dongcarl> But I haven't looked into writing guix import things
15:00 < nothingmuch> or, more precisely - they can only refer to past commits
15:00 < dongcarl> I heard it's not that hard tho
15:00 < nothingmuch> yeah it's pretty sensible from what i've seen
15:00 < nothingmuch> and it doesn't have to literally be guix import, it can just be similar
15:01 < dongcarl> nothingmuch: Hmmm... I'll have to think about that more
15:01 < nothingmuch> but conceptually i think it's more correct to take a bitcoin verified commit and programmatically derive guix package definitions from it
15:02 < dongcarl> Not 100% sure what you mean, but it seems like we have some time till we get there :-)
15:02 < nothingmuch> (by verified i mean as per contrib/verify-commits)
15:03 < dongcarl> nothingmuch: Anything else? :-)
15:04 < nothingmuch> what it boils down to is which repo is considered authoritative for the sha256s that are specified to guix - if you try to put those statically into bitcoin's own repo then there's that circularity issue
15:04 < nothingmuch> nope, gonna cheat with SOURCES_PATH tarball now, and see if i can get a make depends pakcage defn. working
15:04 < dongcarl> Lovely! Keep me posted.
15:05 < nothingmuch> circularity issue = as you add the hashes, the checked out hashes change, unless you make two commits to master each time, once to update sources, and another to update the package definitions to point at that previous commit
15:06 < dongcarl> Ah I think I see what you're saying
15:07 < dongcarl> Eh, sounds like something we can bikeshed later. Worse comes to worst we can always just make another repo
15:08 < nothingmuch> yep. but a good argument for preferring to keep ground truth defined as it currently is, and derive guix package definitions form it
15:08 < dongcarl> yup
15:24 -!- brett-soric [~brett@ppp203-122-220-93.static.internode.on.net] has joined #bitcoin-builds
16:13 -!- brett-soric [~brett@ppp203-122-220-93.static.internode.on.net] has left #bitcoin-builds []
19:01 -!- hebasto [~hebasto@95.164.65.194] has joined #bitcoin-builds
19:01 < fanquake> dongcarl can you concept ack / nack 16270. It's not critical, but at least we can take advantage of the new configure flags to skip building more stuff. I'd probably rather also just not build software with CVEs.
19:01 < fanquake> Also, I was also going to take a look at 16150, and spend some time today testing trimming some packages. Were you looking at that at all?
23:12 -!- Netsplit *.net <-> *.split quits: midnightmagic, wumpus
23:18 -!- Netsplit over, joins: midnightmagic, wumpus
--- Log closed Sun Jul 07 00:00:04 2019