--- Log opened Thu Jun 13 00:00:40 2019 00:24 < wumpus> the guix-built bitcoind shows, at start: 2019-06-13T07:23:53Z Bitcoin Core version v0.18.99.0-unk (release build) 00:24 < wumpus> what's -unk :-) 00:25 < fanquake> 👀 00:25 < wumpus> anyhow two of my nodes (one amd64 one riscv64) are running guix-built executable now, let's see how it goes 00:26 < wumpus> also all testcases from test_bitcoin passed 00:27 < fanquake> nice! 00:27 < fanquake> I'll test out your RISC-V QEMU instructions tonight 00:27 < wumpus> awesome 01:33 < wumpus> I just had the bitcoin-qt hang at shutdown at "GUI: NotifyUnload" and had to kill the process, but cannot reproduce it, I don't know if it's a guix build related issue 01:33 < wumpus> probably not 04:34 < wumpus> ARM32 is ok too 04:42 < wumpus> ARM64 tests pass as well 04:47 < wumpus> dongcarl: thanks for your work on #15277, I don't know what the exact criterion for merge is, but I think we're getting close to the stage where there's no critical problems anymore and we can merge it and then improve later? 04:50 < fanquake> I somewhat agree. It’s also very self contained in /contrib 04:50 < fanquake> I can ACK it a little later 04:51 < wumpus> x86_32 passes too (don't have any 32 bit installs though so tested on 64 bit ubuntu...) 04:52 < wumpus> yes, it's entirely self-contained, just have to be sure there's no regressions for gitian 04:53 < wumpus> in any case, practice has shown that it's easier to get testing for things like this once it's already part of master, and the concept ACK was resoundingly clear on this anyhow 05:34 <@dongcarl> wumpus: Yes I believe we’re close to a point where this can be merged and improved on later. Let me add some documentation. Will read all of your messages once I get back online in an hour! 05:47 < wumpus> agree documentation is very important 05:51 < wumpus> we definitely have tons of weird x dependencies, for one I'd never heard of xtrans 06:38 <@dongcarl> wumpus: Yeah, we should resolve the x dependencies as a separate issue. They're mostly for qt, and I looked closer at the qt logs and it seems to pick up some of the x dependencies and ignore some others... 06:39 <@dongcarl> by the way, reproducibility doesn't work yet, I will upload 2 outputs today, one generated by my Arch machine and one by my Ubuntu machine, and perhaps we can all investigate the diffoscope to see what's wrong 06:39 <@dongcarl> I remember fixing all the obvious sources of non-reproducibility, but the remaining one seems to be more elusive 06:40 < fanquake> Sounds like a plan 06:40 < fanquake> Who's throwing up the bounty :p 06:40 <@dongcarl> Hahaha 06:40 <@dongcarl> Your prize: my eternal gratitude 06:41 < fanquake> That'll do. No doubt you guys will have nailed it down before the AM here anyways hah. 06:41 <@dongcarl> We'll see :p 07:00 < wumpus> that's strange! 07:01 < wumpus> agree that's the next thing to work on then, if it's not deterministic it's kind of pointless :-) 07:06 <@dongcarl> Yeah. I'm running the builds now... It doesn't seem to be a faketime problem, as I did this: "1. build 2. move the directory to `.bak` 3. clean and build again" and that process was deterministic... Perhaps the non-determinism comes from `$PWD`? Idk, but we'll see when I upload the tarballs. 08:03 < kanzure> $PWD because maybe using a $HOME 08:05 <@dongcarl> Yes that might be true 08:09 <@dongcarl> cfields: It seems that we supply the `-qt-xcb` flag to our `qt` build: https://github.com/bitcoin/bitcoin/blob/431d81b61ca968da2d7c25f0d56455a44cd46fed/depends/packages/qt.mk#L100 08:10 <@dongcarl> Ah, so we basically supply our own version of the dependencies listed here that doesn't get provided by `-qt-xcb`? https://doc.qt.io/qt-5.9/linux-requirements.html 11:06 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds 11:24 <@dongcarl> wumpus fanquake: Here's the link for Arch: https://www.dropbox.com/sh/f2gn7xde3016dhs/AABZfhUnoOdMPRvcxqLfuRAia?dl=0 11:24 <@dongcarl> Here's the link for ubuntu: https://www.dropbox.com/sh/5qhp9uf8zdpqfln/AABhUSjrFII11NbnhBMQ_xw-a?dl=0 11:42 < wumpus> thanks, will take a look 11:50 < cfields> dongcarl: finally back and ready to get to work :) 11:50 < cfields> dongcarl: yes. 11:51 < cfields> dongcarl: qt basically does what we do. They build a stub library to satisfy the x runtime deps. 11:51 < cfields> but you have to provide the ones that it can't. 11:52 < cfields> dongcarl: are you good on cctools now? I discussed with fanquake quite a bit in Amsterdam. 11:52 < wumpus> it's kind of crazy that we have to build so much of x, but yes it makes sense 11:53 < cfields> wumpus: even more annoying since they seemingly tried to modularize it (xproto), but the other stuff ends up being required anyway it seems. 11:53 < cfields> I honestly have no clue how it's all glued together. I just worked out the dep chain :) 11:54 < wumpus> no one has a clue what xtrans is it's funny 11:54 < fanquake> dongcarl cheers 11:55 < wumpus> but yes it's modularized :D 11:57 < cfields> heh 11:57 < cfields> Well for the 500th time, I'm happy to have more eyes on this stuff :) 11:57 < cfields> So thanks again fanquake/dongcarl. 11:58 <@dongcarl> I ran a bunch of gitian builds today, seems like my libtool purge made qt fail to pick up XLib and Xcb-XLib 11:58 < fanquake> FYI. Pretty sure weekly meetup about to start in the Core dev channel? 11:58 <@dongcarl> But qt still builds fine... which is weird 11:58 < wumpus> ohnooo 11:58 < wumpus> fanquake:yes 11:58 < cfields> dongcarl: I would expect that you may need the 'pkg-config --static' trick for qt as well. 11:59 <@dongcarl> I’ll fix it. We need to have diffs of configure output for each depends package when we change depends. 11:59 < cfields> I'm still confused about why that isn't needed more (I'm behind on reading discussions, though) 11:59 <@dongcarl> Maybe something for DrahtBot 11:59 < wumpus> wayland builds when ! *ducks* 11:59 < cfields> hah! 11:59 <@dongcarl> Hehe 12:00 < MarcoFalke> Doesn't DrahtBot put those on? 12:00 < MarcoFalke> the build.log 12:00 < cfields> wumpus: last time I checked they didn't allow for switching protos when built as static. 12:00 < cfields> Even though it should be technically possible to just compile in both back-ends, and choose at runtime. Just a matter of missing machinery afaik. 12:00 < wumpus> cfields: I'm pretty sure that's true 12:01 <@dongcarl> MarcoFalke: yeah, but parsing is a little difficult. Could automate config.log diffs (happy to PR myself) 12:01 < wumpus> you either build qt for X, or for wayland, or for another backend, not multiple 12:02 < MarcoFalke> ah good point 12:48 -!- meshcollider [meshcollid@gateway/shell/elitebnc/x-eljktkufsrujivha] has joined #bitcoin-builds 12:48 -!- gertjaap [sid322815@gateway/web/irccloud.com/x-coveqeapfqaqvcce] has joined #bitcoin-builds 12:53 < jb55> dongcarl: a guix CI server for bitcoin would be cool, it could use verified substitutes. nix has hydra, not sure about guix. 12:53 <@dongcarl> jb55: hail: http://hydra.gnu.org/ 12:53 * jb55 hails 12:53 <@dongcarl> Yeah I'll run a substitute server at some point 12:53 <@dongcarl> Low on priority list tho 12:54 < cfields> neat :) 12:54 -!- nothingmuch [~nothingmu@unaffiliated/nothingmuch] has joined #bitcoin-builds 12:54 < jb55> dongcarl: I imagine it would be pretty important for fast CI builds... 12:54 <@dongcarl> jb55: Oh... For fast CI... very good point... 12:54 <@dongcarl> Yeah, that's a little ways off I'd think 12:55 < cfields> One other big thing that can be done for depends is to make it fully parallelizable. 12:55 < cfields> I've started that in the past but never finished. 12:55 < cfields> For ex, qt and boost could build simultaneously. 12:55 < fanquake> Have we got enough non dependant packages that it would speed up a lot? 12:56 <@dongcarl> cfields: Yeah... One of the next steps is to write an importer for our depends syntax for Guix 12:56 < fanquake> I guess OpenSSL as well. 12:56 <@dongcarl> After that... Guix is fully parallel 12:56 < cfields> qt needs openssl 12:56 < fanquake> Ah yea. 12:57 < jb55> dongcarl: do you know if there's a source audit tool for guix? one thing I always wanted was something like diffoscope but for source code when upgrading dependencies. 12:57 * dongcarl will be more heads-down build system after residency is over he promises 12:57 < fanquake> zmq, rapidcheck, libevent 12:57 < cfields> It would be really nice if binutils supported stub libraries like macos 12:58 < cfields> I wonder if they'd be open to it. 12:58 <@dongcarl> jb55: Don't think so, you should ask on #guix tho 12:58 <@dongcarl> cfields: Would that speed up builds? 12:59 < cfields> dongcarl: no, but it would allow us to do new clever things. Like not build at all. 12:59 <@dongcarl> cfields: Not build bitcoin or not build depends? 13:00 < cfields> not build throwaway depends. They'd just be flat text-files listing symbols. 13:00 < fanquake> Oh right. Is this the .tbd stubs 13:00 <@dongcarl> Oh right I remember you talking about that 13:00 < cfields> Yup! 13:01 <@dongcarl> We would need every depends package to support `make package.tbd` tho, right? 13:01 < cfields> dongcarl: it'd be a 5-year-out dream. Wasn't a realistic suggestion for now :) 13:02 < cfields> But no, you'd just do a symbol dump into a flat file. 13:03 <@dongcarl> cfields: Yeah, perhaps good to propose it on mailing list somewhat soon, just to get it in the right minds... 13:03 < fanquake> Damn bitcoin maintainers always trying to improve other people’s build systems 13:04 <@dongcarl> Ikr, it's like we care about reliable software... or something... 13:05 < wumpus> stub libaries would be kind of nice, it seems no one in linux is really interested in them in my experience though 13:06 < cfields> wumpus: that's the impression I've gotten as well. 13:06 < cfields> wumpus: it's nice to see them in the wild for macos, though. Genuinely makes life easier. 13:06 < wumpus> didn't know macos did that now 13:07 < cfields> Though I suspect the motivation was actually to avoid having to distribute binaries as part of their SDKs. Presumably for licensing reasons. 13:07 < cfields> Yup. SDK libs are stubs now. 13:07 < wumpus> yes, there's bascially two reasons to do it, licensing for closed-source software and build parallelism 13:08 < cfields> You mean arch independence? Because that's a neat side-effect too. 13:08 < cfields> Maybe. That's probably pretty complicated in practice. 13:09 < wumpus> at ASML they used it for a latter, when I worked there, it means the library can be built at the same time as the code that depends on it 13:09 < cfields> Aha! 13:09 * cfields office -> home 13:10 < wumpus> but they have some interface definition language; so the stub library can be built from that. I don't think you can do the same with C header files 13:10 < wumpus> (or C++...) 13:11 < wumpus> though your idea of 'a plain text file with symbols and attributes' is basically that :-) 13:11 < jb55> whats an example of a throwaway dep where you would want to do this? 13:11 < wumpus> xtrans ! 13:11 < jb55> excuse me 13:11 < wumpus> (no, I actually don't know, all the X stuff at least) 13:12 < jb55> ah 13:12 < wumpus> ls depends/packages/x* 13:13 < wumpus> those are only needed to build qt 13:14 < wumpus> it's mostly C code isn't it? I'm pretty sure building stub libraries is be easier for that than for C++ 13:14 < wumpus> it gets kind of messy with classes 13:25 < nothingmuch> dongcarl: re importer - is the long term plan to make the deps into guix packages? right now guix is only used for the toolchain parts right? 13:26 < nothingmuch> if yes re first q, how would cross compiled deps look as resources in /gnu/store? 13:27 <@dongcarl> nothingmuch: Guix packages that aren't as link-y as default Guix packages, Guix already knows how to cross compile, they'll just have different hash and end up in a diff directory as their package definition is different 13:38 < wumpus> I guess if you'd want to convert depends to guix, guix packages already exist for many of the deps 13:40 < wumpus> e.g. most of the X stuff is here: https://www.gnu.org/software/guix/packages/X/page/2/ 13:40 < wumpus> ofcourse, qt-static would be something that isn't already available 13:41 < wumpus> but at least for the indirect deps 13:41 <@dongcarl> wumpus: Right that might be doable 13:42 <@dongcarl> Then I can publish depends on substitute servers 13:42 <@dongcarl> faster CI let's go! 13:43 < wumpus> yes! 13:47 * jb55 cheers 14:04 <@dongcarl> cfields: Here's what success and failure looks like when qt tries to check for XLib: https://gist.github.com/dongcarl/a04dfd718d3005410b27cb18c907da5f 14:05 <@dongcarl> It seems that with the libtool files removed, qt doesn't know how to find xcb when `-lX11` 14:05 <@dongcarl> Not sure how we can get pkg-config in here somehow 15:26 < nothingmuch> i'd be happy to look into this, i noticed a few months back that guix import mishandles perl packages' deps, so familiarizing myself with guix import is on my list anyway 15:26 <@dongcarl> nothingmuch: That’d be wonderful. Let me know if you run into any trouble! 15:37 -!- TheCharlatan [~TheCharla@109.236.87.57] has joined #bitcoin-builds 15:42 < TheCharlatan> What is the difficulty of the macOS build beyond adding llvm to the guix toolchain? 15:47 < nothingmuch> wumpus: re packages existing - i believe all of them are available, guix has a bitcoin package (guix edit bitcoin-core to view src) but i haven't verified it is complete. at the very least the bdb 4.8 needs to be added otherwise --with-incompatible-bdb is required 15:50 < nothingmuch> when trying to avoid qt i've also run make check on a few PRs on debian in guix environment --ad-hoc autoconf automake gcc-toolchain libtool pkg-config bdb boost openssl libevent, also needs --with-boost=$GUIX_ENVIRONMENT (describedin pkg src) 20:22 -!- Netsplit *.net <-> *.split quits: wumpus 20:38 -!- Netsplit over, joins: wumpus 22:02 < nothingmuch> https://gist.github.com/nothingmuch/3a785e64ab6f99ba6a4f981ab370c685 --- Log closed Fri Jun 14 00:00:41 2019