--- Log opened Tue Jul 16 00:00:12 2019 04:44 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Ping timeout: 260 seconds] 04:57 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds 06:25 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection] 06:26 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds 07:32 -!- hebasto [~hebasto@95.164.65.194] has joined #bitcoin-builds 08:38 < dongcarl> All my Guix changes have been upstreamed... Need to mess with inferiors and channels now... 08:39 < dongcarl> Also started a spreadsheet for the qt debugging... Should be able to find the problem with systematic log parsing... 08:39 * dongcarl should get the job description of "systematic log parser" 09:32 < jb55> certified grep technician 09:33 < dongcarl> yuuup 09:57 < cfields> dongcarl: here works :) 09:58 < dongcarl> Hello! 09:58 < dongcarl> Okay, so I did some digging... 09:58 < dongcarl> It seems when we cleaned up libtool archives, qt fails to pick up XLib 09:58 < dongcarl> But! It still works... 09:59 < dongcarl> Because xcb is a replacement for XLib (from my research) 09:59 < cfields> Sorry for not chiming in on the deps stuff more. I really want to review it all at once, I just haven't had a chance yet. 09:59 < dongcarl> cfields: No worries at all, we're just glad you're here now :-) 09:59 < dongcarl> So I guess the question is... Can we just ditch XLib? 10:00 < cfields> Mmm. IIRC xcb was intended to be a replacement for XLib, but stuff ends up requiring both... 10:00 < cfields> That's the foggiest, most vague memory ever, though. I'd have to go back and re-learn how it's all glued together. 10:00 < dongcarl> I've been reading from: https://www.x.org/wiki/guide/xlib-and-xcb/ 10:01 < dongcarl> Do you have any recollections of: https://github.com/bitcoin/bitcoin/pull/5915 10:01 < cfields> Ugh, yeah. 10:02 < dongcarl> One sec while I upload some logs 10:02 < cfields> I don't think that should be much of an issue anymore, though. IIRC it was a single binary release that caused issues. 10:03 < dongcarl> cfields: Okay I see... So in the qt configure summary... It seems like it doesn't use XLib anymore after failing to pick it up 10:04 < cfields> And that runs ok? 10:04 < dongcarl> cfields: To the best of my recollection, yes... 10:05 < dongcarl> Here's the configure summary: https://gist.github.com/dongcarl/292ca770b8a78b9d9a5c8cf282c67613 10:05 < cfields> Maybe qt's dependencies have narrowed over the years? 10:05 < cfields> I definitely tried to build it as minimally as possible. 10:06 < dongcarl> This is curious: Using system-provided XCB libraries .. no 10:06 < dongcarl> shouldn't it pick up depends XCB as system-provided? 10:07 < cfields> $(package)_config_opts_linux += -qt-xcb 10:07 < dongcarl> It seems to me that... because we give qt -qt-xcb, we don't even use 10:07 < dongcarl> right... 10:07 < dongcarl> We don't even use the xcb we build then... 10:08 < cfields> I'm afraid you're trying to oversimplify this a bit. 10:08 < cfields> Somewhere there's a doc about how qt-xcb works, sec 10:08 < dongcarl> https://doc.qt.io/qt-5/linux-requirements.html 10:11 < cfields> The '-qt-xcb' configure option can be used to get rid of most xcb- dependencies. Only libxcb will 10:11 < cfields> still be linked dynamically, since it will be most likely be pulled in via other dependencies anyway. 10:11 < cfields> This should allow for binaries that are portable across most modern Linux distributions. 10:12 * dongcarl digesting 10:12 < cfields> https://code.qt.io/cgit/qt/qtbase.git/tree/src/plugins/platforms/xcb/README 10:13 < dongcarl> cfields: Ah! 10:13 < cfields> X is kinda a hydra 10:13 < dongcarl> s/kinda/the definition of/ 10:14 < dongcarl> Okay, so. That made things much more clear 10:14 < cfields> qt basically does what we do, attempts to bake as much in as possible, and require only a minimal abi. 10:14 < cfields> In this case, they still require libxcb at runtime. So you have to build one to link against. 10:15 < dongcarl> cfields: Oh, they dlopen libxcb at runtime? 10:15 < dongcarl> Oh sorry 10:15 < dongcarl> linked dynamically 10:15 < dongcarl> right 10:15 < cfields> They could get by with just the headers if that were the case :) 10:15 < cfields> I assume they can't because X libs get put in all kinds of weird places. 10:15 < dongcarl> Okay so, we should definitely keep libxcb in our tree, correct? 10:15 < cfields> Yep 10:16 < cfields> But I wouldn't have expected it to work without Xlib. I thought it ended up being required for some annoying sync calls or something. 10:17 < dongcarl> Huh... 10:17 < dongcarl> Yeah we can verify that everything works without XLib, are you expecting qt builds to fail or for bitcoin-qt to fail at runtime? 10:19 < cfields> I would have expected that to cost us some features. Most notably, putting stuff on the screen :) 10:19 < cfields> I'll read up more and get you a better answer on that. 10:21 < dongcarl> cfields: Okay cool! So for https://github.com/bitcoin/bitcoin/pull/5915, can we just revert? 10:22 < cfields> Looks like we can if we're comfortable assuming libxcb-xkb < 1.10 is sufficiently deprecated. 10:22 < cfields> Let's see... 10:23 * dongcarl is thankful for the help 10:26 < cfields> Grr, I can't find packages for ubuntu older than Xenial 10:26 < dongcarl> cfields: https://pkgs.org/download/libxcb 10:27 < dongcarl> 14.04 is 1.10 10:27 < cfields> Ok 10:28 < cfields> Mmm, blah, the whole point of doing the qt compat builds is so that they run even on old stuff. 10:28 < cfields> Sec, let me rehash that bug. IIRC it was an include-order thing. 10:29 * dongcarl is thankful 10:29 < cfields> Ok, right, got it. 10:29 < cfields> So, the general problem is... 10:29 < cfields> qt builds its own libxcb-* stuff... 10:30 < cfields> which is just a copy of the libxcb source tree, which they build internally, then add a few wrappers for.. 10:31 < cfields> Problem is, during our configure, we used a big hammer to pass CFLAGS/CXXFLAGS in, which ended up frontrunning theirs, causing qt's internal xcb to be built against depends headers, rather than their internal ones. 10:32 < dongcarl> cfields: Why is that bad? 10:33 < cfields> dongcarl: imagine if you're building bitcoind, and it picks up and builds against /usr/include/validation.h instead of ours. 10:34 < dongcarl> Oh I see...... 10:34 < cfields> It might work, if you copied src/validation.h to /usr/include for some reason. But it's intending to be including an internal file. 10:34 < cfields> So... 10:35 < dongcarl> Okay but don't we want qt to build their libxcb-* stuff against the libxcb we built? 10:35 < cfields> We did that because we used a nasty env var that gcc/clang respected. It was a quick hack. I should've fixed it more cleanly. 10:35 < cfields> dongcarl: this IS their libxcb build. This is all internal... 10:36 < cfields> dongcarl: maybe I should've said upfront, this was a fix for a legit upstream qt bug. 10:36 < cfields> It was just a really nasty/lazy fix. 10:37 < cfields> So it's possible (probable) that it's been fixed upstream anyway. And beyond that, I'm pretty sure we don't use the nasty env var thing anymore, so it probably shouldn't be an issue anyway... 10:37 < cfields> (sorry, braindumping to catch myself back up) 10:37 < dongcarl> No worries, more info is always good 10:37 < cfields> Checking now to see if we still do that 10:40 < dongcarl> cfields: I'm verifying now that the `bitcoin-qt` produced by a build where qt didn't pick up X11 works perfectly, and `ldd` doesn't show `libX11.so` at all 10:40 < dongcarl> also verified that our 0.17.1 release `bitcoin-qt` DID have `libX11.so` in `ldd` output 10:41 < cfields> Neat, ok. 10:41 < cfields> Let's table that for a min though and finish up on the headers thing. 10:41 < dongcarl> Sounds good. 10:42 < cfields> So, can I give you some homework? We can test to see if it's still a problem. 10:42 < dongcarl> Yes, happy to do homework. But I'm still unclear on a few things, so let me phrase my questions... 10:42 < dongcarl> Let me describe what I understand to be true, and just stop me when I'm wrong 10:43 < cfields> Ok, but! 10:43 < dongcarl> but? 10:43 < cfields> Understand that I've never really understood how much of the X stuff is glued together internally. So don't put too much stock in my answers. 10:44 < dongcarl> roger that 10:44 < dongcarl> Okay so we build libxcb in depends as a package 10:45 < dongcarl> And qt builds it own libxcb and libxcb-* 10:45 < dongcarl> and we want them to be separate, correct? 10:46 < cfields> I'm not sure if it builds a copy of what's already been provided or not, but yes, it builds some of the modules internally. 10:46 < cfields> And to speak concretely, the issue is with whether the internal build picks up the internal xcb.h, or the depends one. 10:47 < dongcarl> internal build being the internal builds of the libxcb-* modules? 10:47 < dongcarl> qt internal build* 10:47 < cfields> Sec, let me link you to exactly what it is. 10:48 * dongcarl is happy 10:48 < cfields> https://code.qt.io/cgit/qt/qtbase.git/tree/src/3rdparty/xcb/README 10:49 < cfields> These are the headers in question: https://code.qt.io/cgit/qt/qtbase.git/tree/src/3rdparty/xcb/include/xcb 10:49 < cfields> Now 10:50 < cfields> Without looking further, I can't recall what's supposed to build against the internal headers, and what's supposed to build against the system. 10:50 < dongcarl> Right okay, I think I'm beginning to understand 10:51 < cfields> But judging by the change I made in https://github.com/bitcoin/bitcoin/pull/5915/files#diff-b55d82ee22670dad1b81533652836c45R22 ... 10:51 < cfields> I would guess that qt doesn't want the system headers at all, only the lib to link against. 10:51 < cfields> (Which makes sense from a compat pov) 10:51 < cfields> So, your homework 10:52 < cfields> Is to make sure that qt doesn't use those headers. And if so, be able to explain how/why. 10:52 < dongcarl> headers for...? 10:52 < cfields> xcb.h 10:53 < dongcarl> Okay so I want to make sure that qt doesn't pick up xcb.h? 10:53 < cfields> (I'd just cheat and do `echo "#error foo" >> xcb.h` in the postinstall step :) 10:53 < dongcarl> cfields: postinstall for libxcb.mk, right? 10:54 < cfields> Yes and yes. 10:54 < cfields> To clarify, this is to see if we can revert the substance of #5915. 10:55 < dongcarl> cfields: Why don't we want qt to pick up xcb.h? For compatibility? 10:55 < cfields> dongcarl: imagine if you're building bitcoind, and it picks up and builds against /usr/include/validation.h instead of ours. 10:56 < dongcarl> Okay gotcha. 10:56 < dongcarl> Right because qt is doing its own depends thing 10:56 < cfields> It's intending to pick up headers from https://code.qt.io/cgit/qt/qtbase.git/tree/src/3rdparty/xcb/include/xcb 10:56 < dongcarl> and we don't wanna mess with that 10:56 < dongcarl> Cool, I think I understood :-) 10:57 < cfields> Right. And it would've been fine, except that we passed in a global include path, which ended up frontrunning the internal headers. 10:57 < dongcarl> gotcha 10:58 < dongcarl> cfields: Thanks for the help and patience, once again :-) 10:58 < cfields> Ok, sorry for the chaotic path, but I think we're on the same page. 10:58 < cfields> Basically you just want to make sure that their build is doing a reasonable thing. 10:59 < dongcarl> Yup yup 10:59 < cfields> And to clarify, the proper upstream fix would've been to make sure that all of qt's internal paths were added before the cflags/cppflags/cxxflags. That was the upstream bug that might've been fixed by now. 11:00 < dongcarl> cfields: That's good to know, I'll check that. 13:15 -!- hebasto [~hebasto@95.164.65.194] has quit [Remote host closed the connection] 13:29 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection] 13:29 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds 13:29 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection] 13:32 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds 15:54 < fanquake> ๐Ÿ‘€ 15:54 * dongcarl waves 15:55 < fanquake> Got some back log to read :) 15:55 < dongcarl> most of it is just me being confused lol 15:56 < fanquake> Well if you were confused, good chance Iโ€™m confused too hah. 15:59 < dongcarl> Braindump: It seems that we don't need the LIBRARY part of libX11, but we DO need the INCLUDEs. We can supply qt with `-no-feature-xlib` to achieve this. 15:59 < dongcarl> upstream content: https://bugreports.qt.io/browse/QTBUG-61452?jql=text%20~%20%22xlib%22 16:01 < dongcarl> If we decide to do this, we eliminate XLib from our list of symbols, which is pretty great. 16:01 < dongcarl> We also get to eliminate libXext 16:02 < cfields> Er, if we have a symbol export for XLib then we certainly can't eliminate it :) 16:03 < dongcarl> cfields: Even if everything works fine? 16:04 < cfields> dongcarl: my point was that it _couldn't_ work fine if there was a missing symbol at runtime. 16:05 < dongcarl> Oh sorry I meant we eliminate X11.so from ldd output, which is fine? 16:06 < cfields> Sorry, but specifics matter here... 16:06 < cfields> you mean 'readelf -d' output? 16:06 < cfields> I would expect it to exist in ldd output even without the direct dependency. 16:07 < dongcarl> sec 16:08 < dongcarl> In current master, `ldd` of bitcoin-qt doesn't show libX11.so.6 16:08 < fanquake> Iโ€™ve read through. Going to process then re-read once Iโ€™ve had at least one โ˜•๏ธ 16:08 < dongcarl> the release binaries for 0.17.1 does 16:08 < dongcarl> this is `ldd` not readelf -d 16:09 < dongcarl> readelf -d shows the same actually, `libX11.so.6` in 0.17.1 release, but not in master 16:09 * dongcarl is sorry about saying "symbol" 16:09 < cfields> Heh 16:11 < cfields> Ok, I assume that came from the qt bump, then? Sounds like qt removed the last of their mandatory libX11 calls? 16:11 < cfields> If so, that would all make sense. Used to be a requirement, isn't now. Used to fail to build without it, doesn't now. 16:11 < dongcarl> cfields: I believe so... This post seems to imply we can use `-no-feature-xlib`: https://bugreports.qt.io/browse/QTBUG-61452 16:12 < cfields> Also, if that's the case, then we definitely want to do it this way. You're exactly right, one less runtime assumption. 16:12 < fanquake> ๐ŸŽ‰ 16:13 < dongcarl> cfields: Let me know if you reach the same conclusion with that post, I'm also confirming on #qt... 16:13 < cfields> dongcarl: Yep, looks like you're right. It'd be nice to see it written somewhere. At least a git commit or something... 16:14 < dongcarl> cfields: We still need X11 headers though... Is there an autoconf/automake option to do headers only? 16:14 < dongcarl> also... does that make sense? headers but no lib? 16:14 < cfields> Easy if it's libtool, sec. 16:14 < cfields> Sure 16:15 * dongcarl waits patiently 16:15 * dongcarl is excited about pruning depends 16:15 < fanquake> depends is definitely hitting the gym 16:16 < fanquake> Slimming right down ๐Ÿ‹๏ธโ€โ™‚๏ธ 16:16 < dongcarl> loll 16:16 < cfields> make install-includeHEADERS install-pkgconfigDATA 16:17 < cfields> the latter is obviously only needed if qt uses pkg-config to find it. 16:18 < dongcarl> cfields: I tried with `make install-data` and somehow the final build didn't have bitcoin-qt in it.... -lqminimal failed or something 16:18 < dongcarl> I should experiment more, might be something else 16:18 < dongcarl> tried X11 with `make install-data` I mean 16:19 < cfields> Yeah, install-data won't do it. 16:20 < cfields> That's like icons and examples and stuff IIRC. 16:21 < dongcarl> It did include the headers 16:21 < dongcarl> https://www.gnu.org/software/automake/manual/html_node/The-Two-Parts-of-Install.html 16:23 < dongcarl> lemme try your way 16:24 < cfields> I believe it's libtool though, right? 16:24 < cfields> I'm not sure if the generic targets get mapped to the libtool ones. 16:24 < dongcarl> cfields: I did inspect that the headers and pc files were there... 16:24 < cfields> s/generic/automake, sorry. 16:25 < cfields> Oh, well you could've just said that :) 16:25 < dongcarl> Yeah... Well, I'm running it now with your make targets, let's see what happens 16:25 < cfields> dongcarl: you're sure qt doesn't try a link test? 16:26 < dongcarl> cfields: So that time everything in depends built successfully... But when building Bitcoin, it somehow decided not to build qt 16:26 < cfields> dongcarl: great work hunting all of this down btw! 16:26 < cfields> Oh. I wonder if we have xlib hardcoded in our detection... 16:27 < dongcarl> :-) 16:27 < dongcarl> Hmmm... maybe 16:27 < cfields> Nope 16:27 < cfields> Wait 16:28 < cfields> PKG_CHECK_MODULES([X11XCB], [x11-xcb], [QT_LIBS="$X11XCB_LIBS $QT_LIBS"]) 16:28 < cfields> I'm sufficiently crossed up now, so I'll let you decide if that's a problem :) 16:29 < dongcarl> Also... I'm getting "No rule to make target 'install-includeHEADERS'" for libX11... 16:29 < dongcarl> So it's probably automake instead of libtool? 16:30 < cfields> Hmm, guess so. 16:34 < cfields> Ah, they have a recursive build. It'd be "make -C include install-x11includeHEADERS" 16:34 < cfields> That looks like nonsense though, yours is much nicer. 16:34 < dongcarl> cfields: Eh, mine installs a bunch of nonsense along with the stuff we need 16:35 < dongcarl> I'll get it working with mine first tho, then see the recursive thing... 16:35 < dongcarl> Any documentation on how recursive targets work like this? 16:38 < cfields> There's a good bit written up on recursive make. Basically, you can setup autotools as a bunch of subprojects, or as one large one. Modules make sense, but end up really sucking in practice because they break make's ability to track across segments. 16:38 < cfields> It's enough to just think of it as "make install" basically just calls "make -C subdir" install, for each subdir. 16:38 < dongcarl> Ah right, I remember you still have a PR open that has something to do with that? 16:39 < cfields> src/man/include/etc. 16:39 < cfields> That's a little different. 16:39 < cfields> I converted us to non-recursive ages and ages ago. 16:39 < cfields> May help to see, sec. 16:41 < cfields> https://github.com/bitcoin/bitcoin/commit/65e8ba4dbed519a3be6d497063c3441550c4b3d6 16:42 < dongcarl> Ah I see... 16:42 < cfields> So libx11 works the way that did before. 16:43 < dongcarl> still not building bitcoin-qt... if I only install-data: "configure: WARNING: Could not resolve: -lqminimal; bitcoin-qt frontend will not be built" 16:44 < dongcarl> cfields: I don't think the x11-xcb line you posted has to do with this, does it? 16:44 < cfields> dongcarl: not sure, you'll need to check config.log. 16:45 < fanquake> I recently discovered that bitcoin-qt doesnโ€™t get built if you pass configure it a depends built with DEBUG=1 16:45 < cfields> fanquake: heh, I wonder if there's an oob access in a configure check :) 16:45 < fanquake> It fails with that same problem with lqminimal. 16:45 < cfields> Oh, interesting. 16:46 < fanquake> Info in #16391 16:46 < dongcarl> config.log for mine: https://pastebin.com/1GT83QSw 16:47 < cfields> x86_64-linux-gnu-ld: cannot find -lX11-xcb 16:47 < cfields> x86_64-linux-gnu-ld: cannot find -lX11 16:47 < dongcarl> Right... Okay, I should remove that PKG_CHECK line then 16:48 < cfields> No... 16:48 < cfields> My guess is that qt doesn't propogate the disabled -lX11 to the .pc files. 16:49 < dongcarl> I can check in depends tree? 16:50 < cfields> Yep 16:50 < cfields> grep the qt .pc files for lX11 16:50 < dongcarl> cfields: None of the `lib/pkgconfig/Qt*` files had X11 16:50 < dongcarl> checked with `cat depends/x86_64-linux-gnu/lib/pkgconfig/Qt* | grep X11` 16:51 < cfields> checked share/pkgconfig too? 16:52 < dongcarl> no qt pc files in share/pkgconfig 16:52 < cfields> Hmm, I'm confused then. 16:53 * dongcarl thinking 16:54 < cfields> Ok, I guess you're right? 16:55 < dongcarl> https://www.irccloud.com/pastebin/PKxNQWHm/ 16:56 < dongcarl> might this be the culprit? 16:57 < cfields> It would only be indirectly... 16:57 < cfields> Try removing that line, but I'd kinda expect a new link failure for bitcoin-qt due to missing libxcb. 16:59 * dongcarl trying 16:59 < cfields> Heading out for the day. 16:59 * dongcarl salutes 16:59 < cfields> As of today I finally have IRC up at my office workstation, so I'll be around here more :) 17:00 < dongcarl> Nice! Thanks for all the help today! 17:02 < cfields> Thanks for the good work! 17:03 < fanquake> cfields: Iโ€™m sure that beast will only just be able to handle an IRC client :p 17:03 < dongcarl> :-) 17:03 < cfields> fanquake: I haven't forgotten about the macos stuff, just trying to segment a bit. 17:03 < cfields> fanquake: haha 17:04 < fanquake> cfields: yea no worries, no rush. Iโ€™m not blocked on that. 17:13 < dongcarl> for the record: removing that PKG_CHECK line worked 17:18 < dongcarl> fanquake: Maybe there are other PKG_CHECK lines that might need removing? I need to think about why this made it work though 17:19 < cfields> dongcarl: remember that we can't just remove it, it still needs to work for non-depends builds. 17:20 < dongcarl> cfields: Right. I need to dig into the root cause 17:43 < dongcarl> For the record: looks like if we switch to xorgproto, we can eliminate xproto, xextproto, xtrans, and libX11... 17:43 < dongcarl> See the provides and package contents section: https://www.archlinux.org/packages/extra/any/xorgproto/ 17:43 < fanquake> ๐Ÿ‘€ 17:43 < dongcarl> Ikr? 17:43 < fanquake> That sounds like a big win if it works 17:44 < dongcarl> Too tired to think about if there will be compat issues... But will try sometime soon 17:44 < fanquake> Maybe brain dump into an /issue/ before you crash 17:44 < dongcarl> I'll do it tomorrow, it's not too complicated :-) 17:45 < fanquake> Cool ๐Ÿ‘ 19:48 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Ping timeout: 260 seconds] 19:49 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds 20:25 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Ping timeout: 260 seconds] --- Log closed Wed Jul 17 00:00:13 2019