--- Log opened Thu Aug 08 00:00:34 2019
08:08 -!- hebasto [~hebasto@95.164.65.194] has joined #bitcoin-builds
09:27 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
09:29 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
09:38 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
09:38 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
09:40 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
09:40 -!- hebasto [~hebasto@95.164.65.194] has quit [Read error: Connection reset by peer]
09:41 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
09:42 -!- hebasto [~hebasto@95.164.65.194] has joined #bitcoin-builds
09:50 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
09:50 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:01 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:02 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:08 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:09 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:12 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:14 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:19 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 244 seconds]
10:20 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:23 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:25 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:27 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:28 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:31 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:32 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:33 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:34 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:38 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:40 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
10:45 -!- hebasto [~hebasto@95.164.65.194] has quit [Remote host closed the connection]
10:47 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
10:50 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
11:00 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
11:01 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
12:03 < jonasschnelli> I agree that its useful to run a CI before upload...
12:03 < jonasschnelli> We can still have support for travis or github internal, etc.
12:03 < jonasschnelli> bitcoinbuild.org just don't has the capacity and funds
12:04 -!- aj [aj@cerulean.erisian.com.au] has joined #bitcoin-builds
12:04 -!- fjahr [sid374480@gateway/web/irccloud.com/x-exeyewzysrtpplib] has joined #bitcoin-builds
12:05 < dongcarl> Once I make more progress on windows... We can explore Guix as a CI, it can reproduce environments locally on any machine, and can generate a docker container, or even a bootable USB if you'd like
12:05 < dongcarl> But we need a stop gap as I know Travis has been having problems?
12:06 < jonasschnelli> Guix would probably make sense as a job in a build
12:06 < jonasschnelli> not Guix only
12:06 -!- phantomcircuit [~phantomci@192.241.205.97] has joined #bitcoin-builds
12:06 < jonasschnelli> since its eventually to monotonic
12:06 -!- hanhua [68840050@104.132.0.80] has joined #bitcoin-builds
12:06 < jonasschnelli> But I'd like to see at least guix builds on master merges
12:07 < dongcarl> jonasschnelli: sorry not 100% on what you mean by monotonic
12:08 < jonasschnelli> dongcarl: I'm also unsure. :) But it looks like if everything gets built in Guix, we not proactively testing other build setups
12:08 < jonasschnelli> or maybe I'm wrong
12:08 < dongcarl> jonasschnelli: Oh I see, yeah it can be adapted when we need it to, but anyway I'm just saying it's possible, let's do easier steps first :-)
12:09 < aj> dongcarl: it'll be too reliable :)
12:09 < MarcoFalke> Yeah, we need to test on Ubuntu, since that is our target OS
12:09 < MarcoFalke> Also Windows and Mac
12:09 < dongcarl> jonasschnelli: If you need more compute power for bitcoinbuilds... I can maybe help with that
12:10 < MarcoFalke> dongcarl *buys AMD cpus*
12:10  * dongcarl MarcoFalke knows me too well
12:10 < aj> how much memory do you need for make -j256 though?
12:10 < jonasschnelli> heh...
12:10 < dongcarl> aj: Dunno, guess we should find out lol
12:12 < jonasschnelli> I recognized, when running 6 VMs in parallel, -j4 was slighly slower (or at least not faster) than -j3
12:12 < jonasschnelli> on a. Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
12:12 < jonasschnelli> 64GB ram
12:12 < dongcarl> jonasschnelli: Yeah, probably the scheduler overhead?
12:13 < jonasschnelli> Maybe. Or my VM's are maybe not create efficient enough (ram / cpu / etc.)
12:13 < jonasschnelli> But it's running at an acceptable speed
12:14 < jonasschnelli> If we want to open it for private forks, 2-3 machines more (with each 6[TBD] VM workers) would be required
12:16 < jonasschnelli> However, I first want to make sure my CI can run 2-3 month without any significant issues,... then scale up or integrate closer
12:16 < dongcarl> jonasschnelli: is your CI running something custom? Or does it have some orchestration software already?
12:17 < fanquake> jonasschnelli: are you accepting PRs atm?
12:17 < jonasschnelli> It's using virsh/KVM under the hood. Mostly its just some scripts and a python build server
12:17 < jonasschnelli> fanquake: sure
12:18 < jonasschnelli> https://github.com/jonasschnelli/bitcoin-core-ci (need to push the newest changes though)
12:18 < aj> dongcarl: hm, 2TB per processor, 4TB per server.. seems like enough memory
12:18 < fanquake> I have opened one against the Python script, just having a look at the code.
12:18 < jonasschnelli> some stuff is missing in the repo though (will add soon)
12:18 < fanquake> jonasschnelli: a ok, I’ll rebase after that.
12:34 < phantomcircuit> aj, make -j peaks at about 30GiB
12:34 < phantomcircuit> there's actually not enough parallelism in the builds to use more
12:34  * dongcarl evil laugh
12:34 < aj> phantomcircuit: wow, for real? i thought you'd be able to get 2GB per cpp file...
12:37 < phantomcircuit> aj, yeah it's more like 1GB per compile unit
12:37 < phantomcircuit> iirc the largest is about 1.5
12:44 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-xylbruckdqjsoaki] has joined #bitcoin-builds
12:45 < MarcoFalke> jonasschnelli: Does your ci allow a remote build farm?
12:45 < MarcoFalke> (Assuming someone volunteers to pay for it)
12:46 < jonasschnelli> MarcoFalke: not yet implemented,... but conceptually possible.
12:46 < fanquake> Time for an AMD based build farm in the chaincode closet.
12:47 < MarcoFalke> stonks 📈
12:47 < jonasschnelli> All possible...
12:47 < dongcarl> stonks :chart_with_upwards_trend:
12:47 < MarcoFalke> dongcarl: lol
12:47 < jonasschnelli> What I really like would be some ARM machines
12:47 < jonasschnelli> And,... one thing could also be running Qt in real GUI VMs and screenshot some situations...
12:48 < jonasschnelli> Would even be possible to detect >10% pixel differences against a static reference image or so
12:48 < phantomcircuit> fanquake, hopefully not a literal closet unless they're cool with the paint peeling
12:49 < phantomcircuit> jonasschnelli, i believe the qemu arm emulation on fast x86 servers is in the realm of "useful"
12:49 < jonasschnelli> phantomcircuit: probably,...
12:49 < jonasschnelli> or just x-compile the ARM builds (as we already do) and run the tests on a ARM host
12:49 < jonasschnelli> (Rock64 or so)
12:49 < dongcarl> yeah, I do that with guix builds
12:50 < dongcarl> there were problems with cross-compiled things not running on real machines
12:50 < dongcarl> (I learned about this as I was reading symbol-check.py`
12:51 < dongcarl> Anyway, I'd like to see if there are existing software that can help us along
12:51 < dongcarl> We can spend our efforts polishing the wheel instead of reinventing it perhaps
12:51 < jonasschnelli> Yes. That would be good
12:56 < dongcarl> Drone looks very well-supported: https://github.com/drone/drone
12:56 < dongcarl> Their pricing model is pay for hosting I believe
12:58 < dongcarl> Sorry, I should clarify: everything is open source and upstreamed, but if you don't wanna host yourself, you can pay them to host it
13:06 < phantomcircuit> ideally something with a central job controller so we could "outsource" builds would be nice
13:06 < phantomcircuit> i have a server i'd be happy to do builds on but not so happy to run a web server on
13:06 -!- hanhua [68840050@104.132.0.80] has quit [Remote host closed the connection]
13:09 < MarcoFalke> Thanks dongcarl Looking into drone
13:10 < jonasschnelli> dongcarl: I haven't looked at existing solutions due to experiences I made in the past...
13:10 < MarcoFalke> Seems like they offer free cpu for open source, but also run on your own infrastructure
13:11 < jonasschnelli> The complexity of the CI we need is reasonable low and general open source software usually suffers the problem of beeing captable for too much
13:11 < jonasschnelli> I think promag had some experience with drone
13:12 < dongcarl> MarcoFalke: Good to know
13:13 < jonasschnelli> Looks like drone would not work with qemu (docker only)
13:13 < MarcoFalke> Yeah :(
13:13 < MarcoFalke> Docker only, so can't run windows, mac or freebsd
13:14 < jonasschnelli> I think qemu is the right choice for us
13:14 < jonasschnelli> Often, those CI systems (probably including drone) are optimized for other needs
13:14 < jonasschnelli> Also,... like travis, most come with overdone web-frontends
13:15 < dongcarl> Agree that we should work with tools that suit our needs
13:15 < dongcarl> Yeah, looks like drone is not the one
13:15 < MarcoFalke> Yeah, CI systems tend to go into the direction where you upload your private keys, so they can publish the app on your behalf :(
13:15 < jonasschnelli> I want small and quick stuff... like the bitcoinbuilds ANSI log: https://bitcoinbuilds.org/index.php?ansilog=48f2d4e6-d148-4312-af41-c71501893dbb.log
13:15 < jonasschnelli> MarcoFalke: exactly...
13:16 < dongcarl> jonasschnelli: So it sounds like we're gunna build a robust bash codebase?
13:16 < jonasschnelli> heh...
13:17 < jonasschnelli> I just think we should only build what we need in a minimal non shiny but useful way
13:18 < dongcarl> jonasschnelli: Agreed
13:18 < jonasschnelli> I'm all for reusing... stuff like "aha" for log formatting, "virsh" for VM management (snapshot rollback, etc.), "etc."
13:18 < jonasschnelli> The rest is just glue code. :)
13:19 < dongcarl> yup
13:19 < jonasschnelli> Maybe the way how bitcoinbuilds tracks its builds is a bit hacky
13:19 < dongcarl> I'm kinda surprised the suckless guys or sircmpwn haven't written some dead-simple CI
13:19 < dongcarl> jonasschnelli: I'll take a look at it soon
13:19 < jonasschnelli> Each VM launches a GNU screen instance where the screen log gets written to a NFS share where the host parses the log
13:20 < jonasschnelli> (and the build happends in that GNU instance)
13:20 < jonasschnelli> works very reliable though
13:20 < jonasschnelli> *GNU screen instance
13:20 < dongcarl> jonasschnelli: screen instance... to trick output program into thinking it's an actual terminal?
13:21 < jonasschnelli> Somehow.
13:21 < jonasschnelli> I use "screen" also very often for other stuff
13:21 < jonasschnelli> It looks like the "screen" -> log-to-file approach is an easy way to get full terminal output
13:22 < jonasschnelli> I guess some qemu serial output tool would also do the job...
13:22 < dongcarl> jonasschnelli: Yup, does it preserve escape codes?
13:22 < dongcarl> colored logs lol
13:22 < jonasschnelli> https://bitcoinbuilds.org/?job=4a53c1a1-b4a6-44aa-97d8-5d6442ab3817
13:23 < jonasschnelli> https://bitcoinbuilds.org/index.php?ansilog=4a53c1a1-b4a6-44aa-97d8-5d6442ab3817.log#l3438
13:23 < jonasschnelli> I think it does? At least there are some colors. :)
13:23 < dongcarl> NOICE
13:23 < jonasschnelli> I guess it can't handle the prorgress bar though: https://bitcoinbuilds.org/index.php?ansilog=4a53c1a1-b4a6-44aa-97d8-5d6442ab3817.log#l3069
13:24 < dongcarl> Eh, wtv, don't care too much about the progress bar. Good work so far tho, and I'm glad there's a repo I can see
13:25 < jonasschnelli> reminds me that I should push more recent work...
13:25 < jonasschnelli> It's all in the flux,... but I guess we can make it stable
13:25 < dongcarl> Haha yeah i don't expect this to have 5 nines just yet lol
13:25 < jonasschnelli> and then never say again "oh, too bad travis is not supporting X/Y"
13:26 < dongcarl> \o/
13:26  * jonasschnelli heads to bed
15:31 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-builds
19:06 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-xylbruckdqjsoaki] has quit [Quit: Connection closed for inactivity]
19:51 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
19:52 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
20:00 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
20:01 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
20:18 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-ijsgycygcvfvtbdh] has joined #bitcoin-builds
21:39 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
21:40 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
21:55 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 248 seconds]
21:56 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
22:28 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 244 seconds]
22:29 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
22:50 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
22:50 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
22:57 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
22:58 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
22:58 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-ijsgycygcvfvtbdh] has quit [Quit: Connection closed for inactivity]
23:09 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
23:10 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
23:14 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
23:15 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
23:25 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 258 seconds]
23:28 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
23:30 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
23:31 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
23:37 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Ping timeout: 268 seconds]
23:38 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
23:51 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
23:51 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has joined #bitcoin-builds
23:59 -!- luke-jr [~luke-jr@unaffiliated/luke-jr] has quit [Excess Flood]
--- Log closed Fri Aug 09 00:00:35 2019