--- Log opened Thu Jun 13 00:00:40 2019
00:24 < wumpus> the guix-built bitcoind shows, at start: 2019-06-13T07:23:53Z Bitcoin Core version v0.18.99.0-unk (release build)
00:24 < wumpus> what's -unk :-)
00:25 < fanquake> 👀
00:25 < wumpus> anyhow two of my nodes (one amd64 one riscv64) are running guix-built executable now, let's see how it goes
00:26 < wumpus> also all testcases from test_bitcoin passed
00:27 < fanquake> nice!
00:27 < fanquake> I'll test out your RISC-V QEMU instructions tonight
00:27 < wumpus> awesome
01:33 < wumpus> I just had the bitcoin-qt hang at shutdown at "GUI: NotifyUnload" and had to kill the process, but cannot reproduce it, I don't know if it's a guix build related issue
01:33 < wumpus> probably not
04:34 < wumpus> ARM32 is ok too
04:42 < wumpus> ARM64 tests pass as well
04:47 < wumpus> dongcarl: thanks for your work on #15277, I don't know what the exact criterion for merge is, but I think we're getting close to the stage where there's no critical problems anymore and we can merge it and then improve later?
04:50 < fanquake> I somewhat agree. It’s also very self contained in /contrib
04:50 < fanquake> I can ACK it a little later
04:51 < wumpus> x86_32 passes too (don't have any 32 bit installs though so tested on 64 bit ubuntu...)
04:52 < wumpus> yes, it's entirely self-contained, just have to be sure there's no regressions for gitian
04:53 < wumpus> in any case, practice has shown that it's easier to get testing for things like this once it's already part of master, and the concept ACK was resoundingly clear on this anyhow
05:34 <@dongcarl> wumpus: Yes I believe we’re close to a point where this can be merged and improved on later. Let me add some documentation. Will read all of your messages once I get back online in an hour!
05:47 < wumpus> agree documentation is very important
05:51 < wumpus> we definitely have tons of weird x dependencies, for one I'd never heard of xtrans
06:38 <@dongcarl> wumpus: Yeah, we should resolve the x dependencies as a separate issue. They're mostly for qt, and I looked closer at the qt logs and it seems to pick up some of the x dependencies and ignore some others...
06:39 <@dongcarl> by the way, reproducibility doesn't work yet, I will upload 2 outputs today, one generated by my Arch machine and one by my Ubuntu machine, and perhaps we can all investigate the diffoscope to see what's wrong
06:39 <@dongcarl> I remember fixing all the obvious sources of non-reproducibility, but the remaining one seems to be more elusive
06:40 < fanquake> Sounds like a plan
06:40 < fanquake> Who's throwing up the bounty :p
06:40 <@dongcarl> Hahaha
06:40 <@dongcarl> Your prize: my eternal gratitude
06:41 < fanquake> That'll do. No doubt you guys will have nailed it down before the AM here anyways hah.
06:41 <@dongcarl> We'll see :p
07:00 < wumpus> that's strange!
07:01 < wumpus> agree that's the next thing to work on then, if it's not deterministic it's kind of pointless :-)
07:06 <@dongcarl> Yeah. I'm running the builds now... It doesn't seem to be a faketime problem, as I did this: "1. build 2. move the directory to `.bak` 3. clean and build again" and that process was deterministic... Perhaps the non-determinism comes from `$PWD`? Idk, but we'll see when I upload the tarballs.
08:03 < kanzure> $PWD because maybe using a $HOME
08:05 <@dongcarl> Yes that might be true
08:09 <@dongcarl> cfields: It seems that we supply the `-qt-xcb` flag to our `qt` build: https://github.com/bitcoin/bitcoin/blob/431d81b61ca968da2d7c25f0d56455a44cd46fed/depends/packages/qt.mk#L100
08:10 <@dongcarl> Ah, so we basically supply our own version of the dependencies listed here that doesn't get provided by `-qt-xcb`? https://doc.qt.io/qt-5.9/linux-requirements.html
11:06 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds
11:24 <@dongcarl> wumpus fanquake: Here's the link for Arch: https://www.dropbox.com/sh/f2gn7xde3016dhs/AABZfhUnoOdMPRvcxqLfuRAia?dl=0
11:24 <@dongcarl> Here's the link for ubuntu: https://www.dropbox.com/sh/5qhp9uf8zdpqfln/AABhUSjrFII11NbnhBMQ_xw-a?dl=0
11:42 < wumpus> thanks, will take a look
11:50 < cfields> dongcarl: finally back and ready to get to work :)
11:50 < cfields> dongcarl: yes.
11:51 < cfields> dongcarl: qt basically does what we do. They build a stub library to satisfy the x runtime deps.
11:51 < cfields> but you have to provide the ones that it can't.
11:52 < cfields> dongcarl: are you good on cctools now? I discussed with fanquake quite a bit in Amsterdam.
11:52 < wumpus> it's kind of crazy that we have to build so much of x, but yes it makes sense
11:53 < cfields> wumpus: even more annoying since they seemingly tried to modularize it (xproto), but the other stuff ends up being required anyway it seems.
11:53 < cfields> I honestly have no clue how it's all glued together. I just worked out the dep chain :)
11:54 < wumpus> no one has a clue what xtrans is it's funny
11:54 < fanquake> dongcarl cheers
11:55 < wumpus> but yes it's modularized :D
11:57 < cfields> heh
11:57 < cfields> Well for the 500th time, I'm happy to have more eyes on this stuff :)
11:57 < cfields> So thanks again fanquake/dongcarl.
11:58 <@dongcarl> I ran a bunch of gitian builds today, seems like my libtool purge made qt fail to pick up XLib and Xcb-XLib
11:58 < fanquake> FYI. Pretty sure weekly meetup about to start in the Core dev channel?
11:58 <@dongcarl> But qt still builds fine... which is weird
11:58 < wumpus> ohnooo
11:58 < wumpus> fanquake:yes
11:58 < cfields> dongcarl: I would expect that you may need the 'pkg-config --static' trick for qt as well.
11:59 <@dongcarl> I’ll fix it. We need to have diffs of configure output for each depends package when we change depends.
11:59 < cfields> I'm still confused about why that isn't needed more (I'm behind on reading discussions, though)
11:59 <@dongcarl> Maybe something for DrahtBot
11:59 < wumpus> wayland builds when ! *ducks*
11:59 < cfields> hah!
11:59 <@dongcarl> Hehe
12:00 < MarcoFalke> Doesn't DrahtBot put those on?
12:00 < MarcoFalke> the build.log
12:00 < cfields> wumpus: last time I checked they didn't allow for switching protos when built as static.
12:00 < cfields> Even though it should be technically possible to just compile in both back-ends, and choose at runtime. Just a matter of missing machinery afaik.
12:00 < wumpus> cfields: I'm pretty sure that's true
12:01 <@dongcarl> MarcoFalke: yeah, but parsing is a little difficult. Could automate config.log diffs (happy to PR myself)
12:01 < wumpus> you either build qt for X, or for wayland, or for another backend, not multiple
12:02 < MarcoFalke> ah good point
12:48 -!- meshcollider [meshcollid@gateway/shell/elitebnc/x-eljktkufsrujivha] has joined #bitcoin-builds
12:48 -!- gertjaap [sid322815@gateway/web/irccloud.com/x-coveqeapfqaqvcce] has joined #bitcoin-builds
12:53 < jb55> dongcarl: a guix CI server for bitcoin would be cool, it could use verified substitutes. nix has hydra, not sure about guix.
12:53 <@dongcarl> jb55: hail: http://hydra.gnu.org/
12:53  * jb55 hails
12:53 <@dongcarl> Yeah I'll run a substitute server at some point
12:53 <@dongcarl> Low on priority list tho
12:54 < cfields> neat :)
12:54 -!- nothingmuch [~nothingmu@unaffiliated/nothingmuch] has joined #bitcoin-builds
12:54 < jb55> dongcarl: I imagine it would be pretty important for fast CI builds...
12:54 <@dongcarl> jb55: Oh... For fast CI... very good point...
12:54 <@dongcarl> Yeah, that's a little ways off I'd think
12:55 < cfields> One other big thing that can be done for depends is to make it fully parallelizable.
12:55 < cfields> I've started that in the past but never finished.
12:55 < cfields> For ex, qt and boost could build simultaneously.
12:55 < fanquake> Have we got enough non dependant packages that it would speed up a lot?
12:56 <@dongcarl> cfields: Yeah... One of the next steps is to write an importer for our depends syntax for Guix
12:56 < fanquake> I guess OpenSSL as well.
12:56 <@dongcarl> After that... Guix is fully parallel
12:56 < cfields> qt needs openssl
12:56 < fanquake> Ah yea.
12:57 < jb55> dongcarl: do you know if there's a source audit tool for guix? one thing I always wanted was something like diffoscope but for source code when upgrading dependencies.
12:57  * dongcarl will be more heads-down build system after residency is over he promises
12:57 < fanquake> zmq, rapidcheck, libevent
12:57 < cfields> It would be really nice if binutils supported stub libraries like macos
12:58 < cfields> I wonder if they'd be open to it.
12:58 <@dongcarl> jb55: Don't think so, you should ask on #guix tho
12:58 <@dongcarl> cfields: Would that speed up builds?
12:59 < cfields> dongcarl: no, but it would allow us to do new clever things. Like not build at all.
12:59 <@dongcarl> cfields: Not build bitcoin or not build depends?
13:00 < cfields> not build throwaway depends. They'd just be flat text-files listing symbols.
13:00 < fanquake> Oh right. Is this the .tbd stubs
13:00 <@dongcarl> Oh right I remember you talking about that
13:00 < cfields> Yup!
13:01 <@dongcarl> We would need every depends package to support `make package.tbd` tho, right?
13:01 < cfields> dongcarl: it'd be a 5-year-out dream. Wasn't a realistic suggestion for now :)
13:02 < cfields> But no, you'd just do a symbol dump into a flat file.
13:03 <@dongcarl> cfields: Yeah, perhaps good to propose it on mailing list somewhat soon, just to get it in the right minds...
13:03 < fanquake> Damn bitcoin maintainers always trying to improve other people’s build systems
13:04 <@dongcarl> Ikr, it's like we care about reliable software... or something...
13:05 < wumpus> stub libaries would be kind of nice, it seems no one in linux is really interested in them in my experience though
13:06 < cfields> wumpus: that's the impression I've gotten as well.
13:06 < cfields> wumpus: it's nice to see them in the wild for macos, though. Genuinely makes life easier.
13:06 < wumpus> didn't know macos did that now
13:07 < cfields> Though I suspect the motivation was actually to avoid having to distribute binaries as part of their SDKs. Presumably for licensing reasons.
13:07 < cfields> Yup. SDK libs are stubs now.
13:07 < wumpus> yes, there's bascially two reasons to do it, licensing for closed-source software and build parallelism
13:08 < cfields> You mean arch independence? Because that's a neat side-effect too.
13:08 < cfields> Maybe. That's probably pretty complicated in practice.
13:09 < wumpus> at ASML they used it for a latter, when I worked there, it means the library can be built at the same time as the code that depends on it
13:09 < cfields> Aha!
13:09  * cfields office -> home
13:10 < wumpus> but they have some interface definition language; so the stub library can be built from that. I don't think you can do the same with C header files
13:10 < wumpus> (or C++...)
13:11 < wumpus> though your idea of 'a plain text file with symbols and attributes' is basically that :-)
13:11 < jb55> whats an example of a throwaway dep where you would want to do this?
13:11 < wumpus> xtrans !
13:11 < jb55> excuse me
13:11 < wumpus> (no, I actually don't know, all the X stuff at least)
13:12 < jb55> ah
13:12 < wumpus> ls depends/packages/x*
13:13 < wumpus> those are only needed to build qt
13:14 < wumpus> it's mostly C code isn't it? I'm pretty sure building stub libraries is be easier for that than for C++
13:14 < wumpus> it gets kind of messy with classes
13:25 < nothingmuch> dongcarl: re importer - is the long term plan to make the deps into guix packages? right now guix is only used for the toolchain parts right?
13:26 < nothingmuch> if yes re first q, how would cross compiled deps look as resources in /gnu/store?
13:27 <@dongcarl> nothingmuch: Guix packages that aren't as link-y as default Guix packages, Guix already knows how to cross compile, they'll just have different hash and end up in a diff directory as their package definition is different
13:38 < wumpus> I guess if you'd want to convert depends to guix, guix packages already exist for many of the deps
13:40 < wumpus> e.g. most of the X stuff is here: https://www.gnu.org/software/guix/packages/X/page/2/
13:40 < wumpus> ofcourse, qt-static would be something that isn't already available
13:41 < wumpus> but at least for the indirect deps
13:41 <@dongcarl> wumpus: Right that might be doable
13:42 <@dongcarl> Then I can publish depends on substitute servers
13:42 <@dongcarl> faster CI let's go!
13:43 < wumpus> yes!
13:47  * jb55 cheers
14:04 <@dongcarl> cfields: Here's what success and failure looks like when qt tries to check for XLib: https://gist.github.com/dongcarl/a04dfd718d3005410b27cb18c907da5f
14:05 <@dongcarl> It seems that with the libtool files removed, qt doesn't know how to find xcb when `-lX11`
14:05 <@dongcarl> Not sure how we can get pkg-config in here somehow
15:26 < nothingmuch> i'd be happy to look into this, i noticed a few months back that guix import mishandles perl packages' deps, so familiarizing myself with guix import is on my list anyway
15:26 <@dongcarl> nothingmuch: That’d be wonderful. Let me know if you run into any trouble!
15:37 -!- TheCharlatan [~TheCharla@109.236.87.57] has joined #bitcoin-builds
15:42 < TheCharlatan> What is the difficulty of the macOS build beyond adding llvm to the guix toolchain?
15:47 < nothingmuch> wumpus: re packages existing - i believe all of them are available, guix has a bitcoin package (guix edit bitcoin-core to view src) but i haven't verified it is complete. at the very least the bdb 4.8 needs to be added otherwise --with-incompatible-bdb is required
15:50 < nothingmuch> when trying to avoid qt i've also run make check on a few PRs on debian in guix environment --ad-hoc autoconf automake gcc-toolchain libtool pkg-config bdb boost openssl libevent, also needs --with-boost=$GUIX_ENVIRONMENT (describedin pkg src)
20:22 -!- Netsplit *.net <-> *.split quits: wumpus
20:38 -!- Netsplit over, joins: wumpus
22:02 < nothingmuch> https://gist.github.com/nothingmuch/3a785e64ab6f99ba6a4f981ab370c685
--- Log closed Fri Jun 14 00:00:41 2019