--- Log opened Sat Jun 15 00:00:47 2019
12:30 < nothingmuch> dongcarl: is guix pull into a separate profile for your guix branch the path of least resistance for being able to run your PR?
12:32 <@dongcarl> nothingmuch: Yeah I think so, but haven't tried that... I just clone the guix repo, modify it, `make`, and use `./pre-inst-env guix blah blah` which uses the custom guix I modified
12:33 <@dongcarl> nothingmuch: I'd pull into main profile just cuz I've tested that plenty
12:33 <@dongcarl> I should make it work with inferiors tho
12:33 <@dongcarl> reproducibility first!
12:38 < nothingmuch> the guix branch is meant to be upstreamed, right?
12:38 <@dongcarl> nothingmuch: Right, at some point it will be upstreamed
12:38 <@dongcarl> I have been slowly upstreaming, you can check the Guix git logs
12:41 < nothingmuch> fwiw, once this finally finishes building my plan was to try and take the simplest dep, and try build a static bin w/ against the guix version, if you saw that gist i posted there's a few exact tarball matches so i was thinking of starting with that and adjusting the guix package definition by hand until it more or less matches make depends
12:43 <@dongcarl> nothingmuch: That sounds like a superb plan! There are a lot of pitfalls along the way (I've tried this before), so please let me know if you run into anything
12:44 <@dongcarl> nothingmuch: make sure you're building with `--max-jobs=something`
12:45 < nothingmuch> to ensure parallism related nondeterminism is exposed?
12:45 <@dongcarl> nothingmuch: Oh I thought you were building my branch of Guix?
12:45 <@dongcarl> nothingmuch: or at least `guix pull`ing?
12:46 < nothingmuch> yes, that's still ongoing and althogh i didn't specify i think it's using 2 based on cpu usage and my previous guix daemon config
12:46 < nothingmuch> i thought you meant when i try building deps i should do that
12:46 <@dongcarl> nothingmuch: You should also do that when building deps :-)
12:47 <@dongcarl> Interesting things to look into for you: `guix challenge` and `guix build --check`
12:47 < nothingmuch> i thought guix challenge was just for substitutes?
12:47 <@dongcarl> nothingmuch: Right, but if we end up running a substitute server, people can `guix challenge` against it
12:47 < nothingmuch> ah, indeed
12:48 < nothingmuch> fwiw my motivation to work on this is partly because my laptop is such a beater and i would like substitutes =)
12:48 < nothingmuch> this = deps
12:48 < nothingmuch> the discussion about CI yesterday motivated me to swap this distraction back in
12:49 < nothingmuch> erm, day before yday
12:49 <@dongcarl> nothingmuch: Have you seen: https://www.gnu.org/software/guix/manual/en/html_node/Daemon-Offload-Setup.html
12:51 < nothingmuch> yep, but i have nowhere to offload to at the moment, so can't help me
12:51 <@dongcarl> nothingmuch: Ah hahaha
12:51 <@dongcarl> I see
12:55 < nothingmuch> and really long builds should guilt me into updating my resume instead of staying in denial about having nowhere to offload anything to ;-)
12:56 <@dongcarl> nothingmuch: Do it! :-) But I _do_ recognize that being able to build/build-partially on less beefy machines is a goal to have
13:00 < nothingmuch> yep, i think guix opens up a spectrum as opposed to the somewhat binary choice of gitian builds
13:01 <@dongcarl> nothingmuch: glad to hear :-)
13:01 < nothingmuch> i meant for reproducibility and how accessible it is to people
13:01 <@dongcarl> nothingmuch: yes!
13:01 < nothingmuch> and performance is definitely a parameter there
14:27 <@dongcarl> New diffoscope for people to look over: https://www.dropbox.com/s/z3i7f1hqsunzexa/output-diffoscope.json?dl=1
14:28 <@dongcarl> To view: `diffoscope --text-color=always output-diffoscope.json | less -R`
14:28 <@dongcarl> I believe we're unearthing some determinism differences between distros that were previously hidden
14:50 <@dongcarl> 5:22:50 PM <dongcarl> What's really curious about this is that every differing byte... differs by 8 exactly...
14:50 <@dongcarl> 5:23:21 PM <dongcarl> obviously the filetype difference yielded by `file` is also curious
19:15 < fanquake> 119mb .json 👀
20:09 < fanquake> dongcarl I'm wondering if the diffoscope difference you're seeing with file() might be due to Arch using file 5.37, and Ubuntu (bionic) using file 5.32. Looks like the way gzip output is generated changed in 5.37, see: https://github.com/file/file/commit/3c12eb8d1418181af93abdd466c64dcddf68ce16.
20:10 < fanquake> Could be a combination of changes in file() between the two versions?
20:10 <@dongcarl> Hmmm... Lemme experiment
20:11 < fanquake> when i run file in an arch container, using a different gnu-debug.tar.gz I'm seeing slightly different output again:
20:11 < fanquake> bitcoin-0.18.99-x86_64-linux-gnu-debug.tar.gz: gzip compressed data, max compression, from Unix, original size modulo 2^32 1058037760 gzip compressed data, reserved method, has CRC, was "", encrypted, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 1058037760
20:12 <@dongcarl> $ file {arch,ubuntu}/output/bitcoin-0.18.99-x86_64-linux-gnu-debug.tar.gz
20:12 <@dongcarl> arch/output/bitcoin-0.18.99-x86_64-linux-gnu-debug.tar.gz:   gzip compressed data, max compression, from Unix, original size modulo 2^32 1057894400 gzip compressed data, unknown method, ASCII, encrypted, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 1057894400
20:12 <@dongcarl> ubuntu/output/bitcoin-0.18.99-x86_64-linux-gnu-debug.tar.gz: gzip compressed data, max compression, from Unix, original size modulo 2^32 1057894400 gzip compressed data, reserved method, ASCII, was "", encrypted, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 1057894400
20:13 <@dongcarl> Here's what I'm seeing, and this is with the same `file`
20:13 <@dongcarl> :-(
20:13 <@dongcarl> strange huh?
20:13 < fanquake> hmm ok. Which version of file is that?
20:13 <@dongcarl> File 5.37
20:13 < fanquake> ok
20:14 <@dongcarl> Hmmmm... But that is a very curious clue
20:35 < fanquake> There is also https://salsa.debian.org/reproducible-builds/diffoscope/merge_requests/27/commits, which proposes constraining file versions to 5.33 - 5.36.
20:42 < fanquake> Also, if the output your seeing in the rest of the diffoscope is always off by 8, I wonder if that ties into the "unknown method" vs "reserved method" difference in file output, as that test seems to boil down to checking if a value at a particular byte is either <8 or >8
20:42 <@dongcarl> fanquake: I believe that diffoscope is just using whatever `file` it finds in `$PATH` to inspect the tarballs for their magic bytes
20:43 <@dongcarl> That's true...
20:44 <@dongcarl> I'm starting to think it's a LOCALE problem...
20:44 <@dongcarl> But we explicitly set it...
--- Log closed Sun Jun 16 00:00:45 2019