--- Log opened Mon Jun 24 00:00:51 2019 00:19 -!- Coy9Emard [~Coy9Emard@ns334669.ip-5-196-64.eu] has joined #bitcoin-builds 00:43 < fanquake> Good couple minutes wondering why `make` is broken until I remembered I need to use `gmake` on openBSD 😭 03:57 < dongcarl> fanquake: haha been there done that 06:48 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds 07:26 -!- wumpus2 is now known as wumpus 07:36 < nothingmuch> yesterday's run seemed to end in my laptop overheating and killing itself, didn't make it to riscv, and aarch64-debug tarball was truncated so it was different, but other than that all hashes matched 08:06 < nothingmuch> dongcarl: is there a reason the source tarball is conditionally built n times in build.sh apart from convenience (i.e. setting up all the env variables, etc)? 08:12 < nothingmuch> afaict the only actual dependency for the make dist step isthe gzip wrapper and TAR_OPTIONS 08:14 < cfields> and git, iirc 08:15 < cfields> if so, we should fix that for the hundredth time. 08:16 < nothingmuch> cfields: the bit that figures out the timestamp? if so that's passed to build.sh as an arg by guix-build.sh 08:21 < cfields> nothingmuch: maybe? I'm talking about discovery of the commit/tag. 08:23 < nothingmuch> cfields: no that's separate, and you're right, there's an implicit dep on this behaviour in build.sh, since it runs make dist and then figures out what the tarball's basename is 08:25 < cfields> I should hope that guix builds would point this out by ending up with wonky filenames due to being an unknown release. 08:25 < cfields> Otherwise, git is in the path and env isn't pure :) 08:26 < cfields> (I could be misremembering, though. We've changed the behavior of that script several times) 08:34 < dongcarl> Oh yeah I’m looking to fix those problems too, they made the script much more complicated 08:34 < dongcarl> What’s a good solution? 08:37 < dongcarl> nothingmuch: for the source tarball... what would you do? I wanted it built inside the env since that’ll ensure we have the same gzip and that version 08:37 < dongcarl> tar version* 08:39 < nothingmuch> afaict the 18.99 value is just in configure.ac and gets bumped (see doc/release-process.md) 08:41 < nothingmuch> is there another value that is more like `git describe` output? 08:42 < nothingmuch> dongcarl: i'm thinking of splitting up build.sh into 3 distinct scripts, dist, deps & core, that are still run in the same environment, as an intermediate step to defining guix packages for these 08:43 < dongcarl> You’re right. I’ll add a make target that can print things. 08:43 < dongcarl> nothingmuch: that’s very good 08:43 < nothingmuch> initially i was going to just start with one package, and slice build.sh into the various gnu-build-system phases, but i think that's too ambitious 08:43 < dongcarl> Hahahaha 08:43 < nothingmuch> so i'm going to start instead w/ the trivial build system and these variants of build.sh 08:44 < dongcarl> For sure 08:44 < nothingmuch> so basically to have the outputs directory as just guix named outputs 08:44 < nothingmuch> and then try and integrate it more tightly with the gnu-build-system code, and break deps apart so it's a meta package 08:44 < nothingmuch> btw there's a print target in the depends makefile 08:45 < dongcarl> nothingmuch: oh? I might’ve added that for convenience 08:45 < dongcarl> Lol 08:45 < nothingmuch> ah yes you did, and quite recently 08:46 < dongcarl> nothingmuch: I think you could try to integrate it with my system as you go step by step 08:46 < nothingmuch> fwiw i think that's going to be the path forward for setting guix make flags etc, basically a guix import variant that relies on make -qp or the print-% target 08:46 < nothingmuch> (for the deps) 08:46 < dongcarl> As in, once you have dist done, just put it in the manifest and remove the dist section in build.sh 08:47 < dongcarl> Keep chomping away 08:47 < nothingmuch> yep, that's basically what i'm going for 08:47 < dongcarl> Lovely! 08:47 < nothingmuch> basically build.sh is pretty intricate, and whenever i try and reproduce it by hand i always miss something 08:47 < dongcarl> Hahaha you and me both... 08:47 < nothingmuch> which is why i thought starting off with the "proper" defn. based on gnu-build-system and its phases is likely to be painful 08:47 < dongcarl> It’s already cleaned up a little from gitian... 08:48 < nothingmuch> so breaking build.sh apart but still relying on it seems like a more incremental approach 08:48 < dongcarl> nothingmuch: huh, but each step you just mentioned still seem like separate gnu-builds to me 08:49 < dongcarl> As in `dist` is one configure + one make 08:49 < dongcarl> Same as depends 08:49 < dongcarl> Same as Core 08:49 < nothingmuch> i was thinking of using the trivial system first and just running the specific build.sh variant first 08:49 < dongcarl> But starting with trivial might be much easier to understand 08:49 < nothingmuch> to avoid recreating all your environment setups in guix itself 08:50 < nothingmuch> that's definitely doable, and in fact might be easier for some things (e.g. pulling values out of the manifest) 08:50 < dongcarl> Running the specific build.sh variant? 08:50 < dongcarl> Not sure what you mean by that 08:50 < dongcarl> (Btw I’m just trying to help with the details this is a great idea overall) 08:50 < nothingmuch> split build.sh into build-{dist,depends,core}.sh, or making the actual build steps into a giant case statement 08:51 < dongcarl> Ah right okay 08:51 < nothingmuch> so far i've been plyaing with splitting it, in order to create a minimal version of it for each target 08:51 < dongcarl> Be absolutely sure to look at the default stages in gnu-build-system 08:51 < dongcarl> Promise me that 08:51 < nothingmuch> i have 08:51 < nothingmuch> and that is the ultimate goal 08:51 < dongcarl> Oh okay then you’re good to go 08:52 < dongcarl> Yeah cuz missing shebang patching will really fuck with your head 08:52 < dongcarl> “Huh? What do you mean /bin/sh isn’t there?” 08:52 < nothingmuch> but i tried to start with that a few days ago, and realized i still have too much doubt/uncertainty about how to structure it correctly 08:52 < dongcarl> fo sho 08:53 < dongcarl> Yeah you’re going about it the smart way 08:53 < nothingmuch> (basically i'm just learning about too many new things at once, bitcoin's build system, guix as a tool, guix as alibrary ecosystem, etc ...) 08:53 < dongcarl> Hahahaha, seems like you soak things up fast! 08:55 < nothingmuch> ehhh, dunno about that, conceptually i'm used to all this stuff, it's just a matter of being familiar with the details to avoid the yaks 08:56 < dongcarl> Cool. Well, lmk if you run into anything weird! 08:56 < dongcarl> Very excited about your progress! 08:57 < nothingmuch> once i actually have anything apart from learning outcomes and build hashes i'll be sure to report ^_^ 08:57 < dongcarl> 👍 09:25 < cfields> nothingmuch: the gitian script works like you're describing, I think. 09:25 < cfields> One source tarball for all builds. 09:26 < cfields> I'm not sure why a guix build would spit out binaries for more than one arch, though? I would expect each arch to be independent. 09:27 < dongcarl> cfields: ah we’re not using Guix’s native packaging functionality 09:27 < dongcarl> We’re just using it to build a namespace/cgroup container 09:27 < cfields> mmm, ok, I'll reserve judgement until looking deeper into it :) 09:28 < dongcarl> Cool. I think nothingmuch is working on switching over to the native packaging functionality 09:28 < dongcarl> This was just an easy intermediary step 09:29 < cfields> got it. 10:49 < nothingmuch> fwiw giving up on package with make dist outputs as first goal, i don't see any reasonable path without configure having access to make depends outputs 10:50 < dongcarl> nothingmuch: right. Do depends first 10:50 < dongcarl> Actually, maybe that should be fixed... but we can do that later 10:51 < nothingmuch> any thoughts on what it might take to make that possible? 10:51 < nothingmuch> i only went as far as finding out configure.ac is hardcoded to require boost 10:52 < dongcarl> Ooof 10:52 < dongcarl> Naive thought: a configure flag that’s “dist only”? 10:53 < nothingmuch> without a PR on core i think the most reasonable way to not have dist rely on make depends directly would be adapt the guix bitcoin-core package 10:53 < nothingmuch> i can try and butcher configure.ac to see whether or not such a flag is plausible, but i suspect i'm severely under qualified 10:59 < dongcarl> make dist is super super lightweight I believe 10:59 < dongcarl> So it might make sense to just make it into a shell script 11:00 < dongcarl> I feel like I’ve seen codebases with configure.ac but also a shell script that makes a dist tarball 11:00 < nothingmuch> hmm, that sounds easier 11:00 < dongcarl> Yeah, we can look up what that script is called conventionally 11:04 < dongcarl> Wait... nothingmuch 11:04 < dongcarl> If you’re making Guix packages... use the dist tarball as input 11:04 < dongcarl> That’s how it’s conventionally done, no? 11:05 < nothingmuch> i was going to make the dist package use git-fetch and then just take the tarball from the output directory 11:06 < dongcarl> Well the dist tarball is just a tarball of the git tree, so you might as well just use the git tree? 11:06 < dongcarl> There might be a few subtle differences but probably not that important 11:07 < nothingmuch> actually let's back up a bit and try and figure out what the long term plan for such packages might be 11:08 < dongcarl> K sec 11:08 < nothingmuch> my goal was to preserve the exact outputs of build.sh if i can help it, making a depends package git-fetch is a departure from this 11:09 < nothingmuch> it seems vaguely useful to be able to guix build and get a source dist that's reproducible, and use that as an input, simply because you did the work of making that deterministic 11:12 < nothingmuch> otoh if this is actually used in a gitian builders type process, where people run guix build --check, guix challenge etc, it's not clear to me how they would even get the package definition in the first place 11:13 * dongcarl thinking 11:15 < nothingmuch> i guess the simplest thing is a guix channel under core github org, that contains packages, plus contrib scripts to generate them in the main repo? 11:16 * dongcarl still thinking 11:18 < dongcarl> so we would produce dist tarball from a git checkout? 11:19 < dongcarl> then produce depends from the dist tarball... 11:19 < dongcarl> Then produce core from depends + dist tarball? 11:19 < dongcarl> nothingmuch: is that what you're thinking? 11:20 < nothingmuch> nope, because url-fetch requires that tarball to actually be present somewhere... i think that only really makes sense for release tarballs 11:21 < nothingmuch> otherwise CI would require a tarball to be uploaded per commit in order to have the same package definitions used for releases and unreleased builds 11:22 < dongcarl> nothingmuch: please describe to me what packages you're envisioning, and what their inputs and outputs are 11:23 < dongcarl> (sorry clear communication is hard in text :-/) 11:24 < nothingmuch> in this scenario, a contrib script given a commit ID would spit out 3 package definitions, depends, dist, and core, where depends does a git-fetch, dist has depends as an input for configure (assuming no dist script), and core has both dist and depends as inputs 11:25 < nothingmuch> and the core package would only indirectly refer to the actual source tree hash through the depends package, whereas the dist package would have the same git-fetch as the depends package 11:25 < nothingmuch> s/through the depends package/through the dist package/ 11:26 < nothingmuch> with make dist factored out to no longer need configure, the dist package could be decoupled from depends package 11:26 < dongcarl> MAKE SENSE 11:26 < dongcarl> Very nice 11:26 < dongcarl> s/MAKE/MAKES/ 11:27 < dongcarl> nothingmuch: Take a look here: https://www.gnu.org/software/guix/manual/en/html_node/Package-Transformation-Options.html#Package-Transformation-Options 11:27 < nothingmuch> and alternatively with url-fetch both depends & core could just name the source tarball ahsh 11:27 < dongcarl> `--with-git-url=` + `--with-commit=` would simplify your "contrib script" 11:28 < nothingmuch> fwiw i was imagining something similar to your gcc derivations, i.e. a guile module that derives package definitions 11:28 < nothingmuch> derivations is not the right word to use here, sorry 11:29 < dongcarl> nothingmuch: Ah right, but I think using the command line flags might make it easier (and less to maintain) 11:29 < dongcarl> Anyway, these are just details 11:29 < nothingmuch> yes, i missed --with-source 11:29 < dongcarl> Yeah that might simplify some steps 11:30 < dongcarl> The plan sounds good. And I can't wait to hear about your progress soon :-) 11:36 < nothingmuch> btw re extracting a dist making script - distdir-am relies on $DISTFILES, which relies on some automake set vars 11:36 < dongcarl> :-/ 11:36 < dongcarl> Maybe a configure flag is what we need then 11:37 < nothingmuch> well, automake can run without configure, so it might just be a question of processing the Makefile.in with a bare bones config.status 11:38 < dongcarl> Kk, lmk what you end up deciding! 11:38 < nothingmuch> or i need to learn how it actually processes the *.in files (is @FOO@ m4?) 11:38 < dongcarl> cfields might be the person to ask there 😊 11:43 < cfields> Not sure what you're after, sounds like you'd have to rewrite a good chunk of the buildsystem to accomplish what you're after. 11:44 < dongcarl> cfields: I think he just wants to be able to `make dist` without building depends first. Is that hard? 11:44 < cfields> without rewriting, yes. 11:45 < cfields> Well 11:45 < cfields> You could potentially add a --no-dependencies flag, but that would make configure.ac a mess. 11:45 < cfields> Hmm 11:46 < cfields> Actually 11:46 < dongcarl> Yeah I was thinking that would be the case 11:46 < cfields> the libbitcoinconsensus build requires no dependencies iirc. 11:47 < dongcarl> Yeah might be good to take a look at that for inspiration. Thanks cfields! 11:54 < cfields> Hmm, yes, I think my knee-jerk was wrong. After some thought, I think that might work. 11:54 < cfields> (After some fixups, I'm sure) 11:55 < dongcarl> Yay! 11:56 < dongcarl> So just emulate what libbitcoinconsensus does? 11:59 < nothingmuch> grep xnono configure.ac suggests some --disable flags, trying it out now 12:17 < nothingmuch> ok, this gets quite close: ./configure --disable-wallet --without-wallet --without-utils --without-daemon --without-qt --disable-bench --disable-tests --without-boost 12:17 < nothingmuch> the only problem is that -lssl and -lcrypto or the corresponding headers (depending on use_pkgconfig) are still unconditionally checked for 12:21 < nothingmuch> hmm, but now make distdir in univalue and libsecp256k1 fails 12:21 < nothingmuch> sorry, just univalue 12:33 < nothingmuch> adding openssl to the manifest to workaround the -lssl check, and running ./configure in src/univalue produces a different tarball hash 12:33 < nothingmuch> unfortunately i screwed up git clean -xdi and zapped my deps, so i'll have to diffoscope this tomorrow 12:36 < cfields> My Univalue integration PR will fix that. 12:37 < cfields> Blah the non-pkgconfig path should just be removed. 12:37 < cfields> And have Windows require pkg-config. 12:40 < cfields> nothingmuch: fyi, you can set BASE_CACHE to somewhere outside of the build dir, then a git clean won't wipe it. 12:40 < cfields> The results are entirely deterministic, there should be no way for a leftover file to screw anything up. 12:41 < nothingmuch> thanks, that's good to know 14:11 < nothingmuch> WORK_PATH too, to cache downloads 14:12 < nothingmuch> (not sure if that can affect build results though, just that setting BASE_CACHE doesn't protect downloads from git clean) 15:31 < nothingmuch> so the difference in the outputs is that the man pages for the disabled tools are missing from doc/man 16:28 < nothingmuch> it's probably something stupid, but i'm at a loss.. removing the conditionalization from doc/man/Makefile.am adds these to the dist tarball, but i can't figure out why, the way i'm reading the code i thought that should only affect install? 16:29 < nothingmuch> i can't see anything in the distdir related rules that seems to pay attention to this 16:42 < nothingmuch> erm, nevermind, for some reason i managed to convince myself DISTFILES did contain those, but i think i was looking at a stale makefile 16:42 < nothingmuch> (printing the var from the actual build script confirms their absence) 16:55 < nothingmuch> https://gist.github.com/nothingmuch/62ca212bbed47890d5a24524981dcc2d # this modified build.sh only produces the source tarball without requiring make depends first 16:58 < nothingmuch> blech, i wish for once i could write a comment without just trailing off mid sentence =( 17:00 < nothingmuch> dongcarl: so anyway, based on above i think i'm going to pivot back to a package that outputs the source tarball as an input for both a depends and a core package 17:03 < nothingmuch> also, would you guys consider --enable/--with etc affecting the distdir target a bug? to me it makes more sense that those would only affact the install target 18:08 -!- ryanofsky [~russ@jumpy.yanofsky.org] has quit [Quit: ZNC 1.7.3 - https://znc.in] 22:11 -!- jonatack82 [25a5605d@37.165.96.93] has joined #bitcoin-builds 22:20 < fanquake> dongcarl going to merge your xtrans changes shortly, just booting Travis into the green 23:41 -!- jonatack82 [25a5605d@37.165.96.93] has quit [Quit: Ping timeout (120 seconds)] --- Log closed Tue Jun 25 00:00:52 2019