--- Log opened Fri Dec 11 00:00:43 2020
00:17 < jonasschnelli> macOS gitian build on master seems to fail: https://bitcoin.jonasschnelli.ch/gitian/build/359
00:17 < jonasschnelli> Fetching v2.1.1.tar.gz from https://github.com/al45tair/mac_alias/archive/ (stalls)
00:33 < wumpus> achow101: I noticed that the sqlite wallet backend recreates the prepared queries for every batch, is there a specific reason for this? I guess it has to do with multi-threading?
00:34 -!- mj_node [7a001982@122.0.25.130] has joined #bitcoin-core-dev
00:35 < mj_node> Folks, I was 95% done syncing my node, and unplugged my external SSD accidently, I tried to restart bitcoind, but it's resuming from block 0, can anyone help? I tried to re-builld, etc.. the [blocks] folder has all the raw blk.dat but I just cant get it to use them. 
00:36 < sipa> mj_node: run with -reindex
00:37 < sipa> it'll still restart the validation from 0, but it won't redownload everything
00:41 < wumpus> normally unless a really high dbcache value is used it shouldn't go back that far on a crash, though unplugging storage while running can cause all kinds of corruption so it's hard to say-in any case a reindex is all you can do, try to keep the device plugged this time :-)
00:42 < mj_node> I tried that -reindex doesnt work, its started to re-download blocks, I launched again re-index it spent hours going through the blocks but still says "loadBlockIndexDB: last block file = 136", when I have over 2000 block files
00:42 < wumpus> ok, wipe everything and start over then
00:42 < mj_node> somehow I think block\index content is the problem
00:42 < mj_node> Im on 1mbps conneciton ,ehehe
00:42 < wumpus> it's corrupted beyond repair
00:42 < mj_node> ouch
00:43 < mj_node> just to understand, is it because the blk.dat files are unique/
00:43 < mj_node> and custom to each ?
00:44 < sipa> mj_node: presumably when it started redownloading it started overwriting the block files you already had
00:45 < mj_node> sipa: yes exactly that's what happened, so even a command like 'reconsider block' doesnt work?
00:45 < wumpus> not necessarily, they contain public information after all, though the blocks don't come in in sequental order so they'll be in different orders in the files on different nodes, the "block index" database contains pointers to where every block is
00:46 < mj_node> @wumpus hmm so surely I should be able to rebuild the block index from my raw blocks on the SDD no?
00:46 < wumpus> yes, that's what reindex does
00:46 < sipa> mj_node: redownloading corrupted the blocks you already had
00:46 < wumpus> but not if the first blocks were overwritten as sipa says
00:46 < mj_node> got it -thanks guys, appreciate the explaination, another 20 days i guess.
00:46 < sipa> so you have a few good blocks at the beginning, then overwritten garbage, and then all old good blocks
00:47 < sipa> 20?!
00:47 < mj_node> slow wifi....
00:47 < sipa> what hardware is this?
00:47 < sipa> ow
00:47 < mj_node> ehehhehe, I'm in quarantine with shitty connection
00:47 < sipa> if network is the bottleneck i can't help you
00:48 < sipa> mj_node: here is something you couldnl try (it's a moonshot, though)
00:48 < wumpus> you might want to copy the block files from someone else on physical storage
00:48 < sipa> start downloading again in another directory
00:48 < sipa> until you have the first few block files (as many as you possibly had overwritten)
00:49 < wumpus> in any case, this isn't a support channel, unless you're doing development and having questions about the code for that reason this is not the place, use #bitcoin next time
00:49 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Ping timeout: 240 seconds]
00:49 < sipa> then copy those over the same-named ones in your real dir
00:49 < sipa> and then do a reindex
00:49 < sipa> also, yeah, what wumpus said
00:49 < mj_node> @wumpus sorry for that, and I will make sure to go to #bitcoin-core-dev
00:50 < mj_node> #bitcoin i mean...
00:50 -!- Pavlenex [~Thunderbi@185.245.85.251] has joined #bitcoin-core-dev
00:50 -!- mj_node [7a001982@122.0.25.130] has left #bitcoin-core-dev []
00:57 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has joined #bitcoin-core-dev
01:13 -!- da39a3ee5e6b4b0d [~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
01:13 < wumpus> support for rescanning with missing block files, and in general for disjointed block directories and downloading only missing blocks would be an interesting feature, and likely required for more eleborate pruning strategies, but I think there's a few things prventing this from working right now
01:20 < vasild> the way blocks and undo are stored now in blk and rev files would make this messy
01:20 < vasild> put everything in sqlite!
01:21 < wumpus> I don't think the storage format is the problem, I think this kind of corruption is easier to handle the simpler your data format is
01:21 -!- jeremyrubin [~jr@c-73-15-215-148.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
01:22 < wumpus> heh :)
01:23 < wumpus> ideally it would get the longest headers chain from P2P *then* start reconstructing
01:23 < wumpus> it's easier to puzzle what fits where then
01:24 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
01:24 < bitcoin-git> [bitcoin] MarcoFalke pushed 5 commits to master: https://github.com/bitcoin/bitcoin/compare/736eb4d80838...6a4806367177
01:24 < bitcoin-git> bitcoin/master 91d6195 Suhas Daftuar: Simplify and clarify extra outbound peer counting
01:24 < bitcoin-git> bitcoin/master 3cc8a7a Suhas Daftuar: Use conn_type to identify block-relay peers, rather than m_tx_relay == nul...
01:24 < bitcoin-git> bitcoin/master daffaf0 Suhas Daftuar: Periodically make block-relay connections and sync headers
01:24 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
01:24 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
01:24 < bitcoin-git> [bitcoin] MarcoFalke merged pull request #19858: Periodically make block-relay connections and sync headers (master...2020-08-blocks-only-rotation) https://github.com/bitcoin/bitcoin/pull/19858
01:24 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
01:30 < wumpus> instead of having 'import blocks' as a discrete initialization phase, consider the unplaced but known blocks already on disk as another block source like P2P (thinking of it I suppose it *almost* works that way already, after reindex-chainstate two-phase reindex process)
01:31 < wumpus> it'd be a lot of un-fun fiddling around with broken block directories and such to develop and test this, doesen't sound like fun :)
01:33 -!- promag_ is now known as promag
01:33 < promag> cfields: ping?
01:40 < wumpus> at the least it'd need to rescan *all* `blk*` files, not stop when one is missing and delete the rest, e.g. get rid of this sledgehammer: https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L658
01:43 -!- jeremyrubin [~jr@c-73-15-215-148.hsd1.ca.comcast.net] has quit [Ping timeout: 260 seconds]
01:46 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection]
01:49 < gwillen> wumpus: one thing that occurred to me, when I had a corrupted block directory myself and was waiting to see if core could recover (it could not), is that you have to be slightly careful
01:50 < gwillen> you don't want any risk of ending up in a situation where your block files on disk actually contain two slightly different copies of the same block
01:50 < gwillen> and you checked one of them, but one or more indices ends up pointing at the other one
01:51 < gwillen> this seems like the sort of thing that could slip in, by being more liberal in what one accepts from the block files
01:52 < gwillen> (especially considered that it is Generally Regarded As Safe to grab this stuff from someone else to bootstrap, trusting that core will check it on startup before using it)
01:53 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Ping timeout: 240 seconds]
01:54 < aj> isn't that what `-loadblock=<file>` is for?
02:02 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
02:02 < bitcoin-git> [bitcoin] Psyruss77 opened pull request #20623: Create msbuild.yml (master...master) https://github.com/bitcoin/bitcoin/pull/20623
02:02 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
02:03 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
02:03 < bitcoin-git> [bitcoin] fanquake closed pull request #20623: Create msbuild.yml (master...master) https://github.com/bitcoin/bitcoin/pull/20623
02:03 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
02:05 -!- kexkey_ [~kexkey@static-198-54-132-93.cust.tzulo.com] has quit [Ping timeout: 260 seconds]
02:12 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has joined #bitcoin-core-dev
02:16 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
02:49 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
02:49 < bitcoin-git> [bitcoin] jnewbery opened pull request #20624: net processing: Remove nStartingHeight check from block relay (master...2020-12-remove-starting-height) https://github.com/bitcoin/bitcoin/pull/20624
02:49 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
02:54 < aj> jnewbery: is the logic for 20624 right? we never go back into IBD without stopping and restarting, but could conceivably get more than two weeks behind (eg if we've got an unattended node whose network was down)
02:59 < jnewbery> aj: potentially, yes. We could get out of IBD, lose connectivity and fall behind, and then start catching up again and relaying old blocks to peers. That's mostly also the case now (the 2000 blocks behind nStartingHeight won't stop us from relaying old blocks to peers if we've been connected to them for a while)
03:00 < aj> jnewbery: presuming we were connected a while, they should be in the same state as us (either both current, or both out of date), so relaying is probably okay; if the network was down, when it came back up, we'd reconnect and choose new starting heights though?
03:01 < jnewbery> I think the worst case is this: we get out of IBD, lose connectivity and fall behind, and then connect to new peers that are on the best tip. We start catching up and then relay headers to peers that are ahead of us. The worst case is 80 bytes for each header, and that peer wouldn't download the block.
03:01 -!- belcher [~belcher@unaffiliated/belcher] has joined #bitcoin-core-dev
03:02 < aj> jnewbery: i guess my impression is that if this works okay if you fall behind, it should work okay if you're still in IBD?
03:04 < jnewbery> the 'this' working ok being 'checking that you're within 2000 blocks of the peer's starting height'?
03:05 < aj> jnewbery: jnewbery: "sending headers, relaying blocks" -- if it's okay to do when you've been out of IBD but are 2001 blocks behind, it should also be okay if you're in IBD?
03:06 < aj> jnewbery: (the "?" there is doing a lot of work...)
03:07 < jnewbery> maybe. I think it's best that we just avoid relaying any inventory when we're in IBD
03:08 < aj> jnewbery: so i think the argument against doing it in IBD is you could be on a relatively low-work but high-height fork from genesis, and end up spamming lots of headers/blocks and waste lots of b/w before everyone realises there's a higher-work fork and switches to that
03:10 < jnewbery> I
03:11 < aj> jnewbery: doing a high-height fork from 2000 blocks ago is expensive, though; and maybe finishing IBD generally is good enough there -- worst case there's a long, low-work chain from a year or two ago, _and_ you've been running but disconnected for that time, but the number of nodes doing that are going to be small, even compared to the number of nodes syncing from genesis at any point in time?
03:11 < jnewbery> I'm confused. Are you arguing against the change in 20624?
03:11 < aj> jnewbery: no, i'm trying to understand it
03:14 < jnewbery> I think worst worst case is you send headers from that long chain from a year or two ago. Two years of headers is ~8MB
03:15 < sdaftuar> aj: in practice, if you're out of IBD, you'll be syncing headers with all your peers shortly after the connection is established
03:15 < sdaftuar> aj: because the criteria for doing that lines up pretty closely (iirc). While in IBD, we purposely sync headers from only one peer to avoid duplication.
03:16 < sdaftuar> so i think if we are announcing blocks to peers while catching up after coming back online (so IBD is false, but we're in fact behind), it's no big deal, because we ought to know our peers' header chain anyway
03:16 < sdaftuar> and therefore won't actually be announcing to them, since we'll know they know the blocks already
03:16 -!- filchef [~filchef@89.253.179.83] has joined #bitcoin-core-dev
03:17 -!- filchef [~filchef@89.253.179.83] has quit [Client Quit]
03:18 < sdaftuar> i guess it's worth testing that there's not some slippage at the beginning of a connection, if they are slow to respond to our getheaders and we are connecting blocks, maybe we'd blast them with useless data?  not sure how likely that is
03:18 -!- Neoma33Crooks [~Neoma33Cr@static.57.1.216.95.clients.your-server.de] has joined #bitcoin-core-dev
03:20 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 264 seconds]
03:20 -!- provoostenator [~quassel@provoostenator.sprovoost.nl] has quit [Remote host closed the connection]
03:20 < jnewbery> sdaftuar: are you talking about after removing the nStartingHeight check? I think if we're out of IBD, we can't be connecting that many blocks?
03:21 < sdaftuar> (just realized "while in ibd, we purposely sync headers from only one peer" is a bit imprecise -- what i meant was that we wait until we have completed header sync from one peer before syncing with others, so we could be connecting blocks whiel still doing header download, and not yet have tried syncing headers with our other peers)
03:21 -!- provoostenator [~quassel@provoostenator.sprovoost.nl] has joined #bitcoin-core-dev
03:21 < sdaftuar> jnewbery: yes i'm talking about that check (well your PR), trying to explain the IBD vs nonIBD distinction better
03:23 -!- Neoma33Crooks [~Neoma33Cr@static.57.1.216.95.clients.your-server.de] has quit [Ping timeout: 256 seconds]
03:23 < sdaftuar> hm! the behavior around exactly how we starting syncing headers witha  peer is not quite what i remember.
03:25 < sdaftuar> it still may not matter much, but it looks like if we fall behind a bunch (say because we lost our network for a while, and then it came back up) that we would only sync headers from one peer until our tip is close to current, before syncing with new ones
03:28 < sdaftuar> anyway, i think that's a long-winded way of me thinking that not-relay in IBD is the right behavior, and relaying while not in IBD even if we happen to be a bit behind is probably ok.
03:30 < jnewbery> I need to understand the "start block sync" and FindNextBlocksToDownload() logic better
03:32 < sdaftuar> the basic idea is that when we start up, we pick a first peer to start syncing headers from. as soon as we have headers that indicate there is a tip >= work of our current tip, we start downloading towards it from any peer that has it.  also, once our headers chain is close to current (time within a day of current time), we sync headers from all peers.
03:33 < sdaftuar> and then as those peers respond with their headers (which should be quick, if our headers chain is the correct one -- a single header with their best tip is typical) we'll download blocks from them as well, since we'll know they have the blocks we need.
03:36 < jnewbery> That certainly makes sense conceptually. I just gind that mapping that design to the various bits of logic and state in SendMessages() and elsewhere is a bit tricky
03:38 < aj> jnewbery: "gind" ?
03:38 < aj> find
03:39 < jnewbery> *find
04:00 < michaelfolkson> I'm looking at bech32 https://bitcoin.stackexchange.com/questions/100508/can-you-break-down-what-data-is-encoded-into-a-bech32-address
04:00 < michaelfolkson> bech32 addresses start with bc representing Bitcoin and then 1 which is just a separator
04:00 < michaelfolkson> But regtest bech32 start bcrt
04:01 < michaelfolkson> Does anyone know what the `rt` represent?
04:01 < wumpus> regtest
04:02 < michaelfolkson> Huh, that was easy. Why does regtest have a rt separator yet testnet and signet both start tb1 an don't need a separator?
04:02 < wumpus> this comes from https://github.com/bitcoin/bitcoin/blob/master/src/chainparams.cpp#L446
04:03 < aj> michaelfolkson: 1 is always the separator in bech32
04:04 < wumpus> both testnet and signet start with 'tb' i don't know why it was chosen to use the same there, probably because they are both test networks and there can potentially already be many of them
04:04 < michaelfolkson> Oh so regtest always starts bcrt1?
04:04 < michaelfolkson> bcrt is human readable and 1 is the separator
04:04 < wumpus> yes
04:05 < michaelfolkson> Ok thanks
04:06 < michaelfolkson> This is the context on why testnet and signet both start tb https://github.com/bitcoin/bitcoin/pull/18267#discussion_r491150895
04:06 < wumpus> this is the best reference of course: https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki
04:07 < wumpus> michaelfolkson: thanks for looking it up so it was as i guessed
04:08 < michaelfolkson> Presumably non-default signets will be encouraged to not start tb
04:08 < michaelfolkson> Although can't force them
04:08 < aj> michaelfolkson: nah, that would require patching the code
04:08 < michaelfolkson> Ohhh non-default signets will also start tb?
04:08 < aj> michaelfolkson: there used to be config options for it, -signet_hrp= or so
04:09 < wumpus> seems from that discussion that all new test networks start with tb, as it's for testing, address overlap is not critical
04:09 < wumpus> that regtest has its own is a historical artifact then
04:09 < aj> yeah, and makes it easier to port wallets to different testnets
04:10 < wumpus> exactly
04:10 -!- peterrizzo [~peterrizz@ool-44c18924.dyn.optonline.net] has joined #bitcoin-core-dev
04:10 < aj> meanwhile regtest is kind-of bitcoin-core specific, and there's no point having wallets work with it
04:10 -!- mj_node [~mj_node@122.0.25.130] has joined #bitcoin-core-dev
04:11 < michaelfolkson> Yeah treating regtest differently to testnet, signet makes sense as not meant to be a public network
04:11 < michaelfolkson> Though choosing tb versus bcrt is a bit peculiar
04:11 < aj> i noticed today that apparently when p2sh was being deployed, regtest didn't even exist
04:12 < wumpus> the public signet is a public network
04:12 < michaelfolkson> And presumably the non-default signets would also be public(ish) networks
04:12 < michaelfolkson> Otherwise use regtest
04:13 < wumpus> yes definitely easier to share
04:15 -!- da39a3ee5e6b4b0d [~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d] has joined #bitcoin-core-dev
04:16 -!- mj_node is now known as symbiosis
04:17 -!- da39a3ee5e6b4b0d [~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d] has quit [Remote host closed the connection]
04:17 -!- gac410 [~gac410@178.162.212.214] has quit [Remote host closed the connection]
04:17 -!- da39a3ee5e6b4b0d [~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d] has joined #bitcoin-core-dev
04:20 -!- Victor_sueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev
04:22 -!- kristapsk [~KK@gateway/tor-sasl/kristapsk] has quit [Remote host closed the connection]
04:22 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Read error: Connection reset by peer]
04:22 -!- kristapsk_ [~KK@gateway/tor-sasl/kristapsk] has joined #bitcoin-core-dev
04:22 -!- ghost43_ [~daer@gateway/tor-sasl/daer] has joined #bitcoin-core-dev
04:23 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Ping timeout: 240 seconds]
04:29 -!- Pavlenex [~Thunderbi@185.245.85.251] has quit [Ping timeout: 240 seconds]
04:37 < jonatack> "// Download if this is a nice peer, or we have no nice peers and this one might do."
04:37 < jonatack> love some of these comments :)
04:37 -!- AmberJ_ [~AmberJ_@178.239.168.171] has joined #bitcoin-core-dev
04:43 -!- dr-orlovsky [~dr-orlovs@31.14.40.19] has joined #bitcoin-core-dev
04:46 < wumpus> it's cute
04:58 -!- da39a3ee5e6b4b0d [~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
05:09 < michaelfolkson> What you looking at jonatack?
05:10 < jonatack> michaelfolkson: grep the codebase ;)
05:11 < michaelfolkson> So demanding of my typing fingers
05:12 -!- kali1 [~kali@i16-les01-ntr-31-36-36-40.sfr.lns.abo.bbox.fr] has joined #bitcoin-core-dev
05:12 -!- kali1 [~kali@i16-les01-ntr-31-36-36-40.sfr.lns.abo.bbox.fr] has left #bitcoin-core-dev []
05:17 < jonatack> you'd be forgiven for thinking that line was from a jane austen novel instead
05:21 < aj> "It is a truth universally acknowledged, that a high-bandwidth archive node must be in want of an inbound connection" ?
05:24 -!- symbiosis [~mj_node@122.0.25.130] has quit [Quit: Leaving]
05:25 < vasild> "uhoh, different"
05:30 -!- da39a3ee5e6b4b0d [~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d] has joined #bitcoin-core-dev
05:34 -!- Guyver2 [Guyver@guyver2.xs4all.nl] has joined #bitcoin-core-dev
05:35 -!- vasild_ [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-dev
05:35 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Disconnected by services]
05:35 -!- vasild_ is now known as vasild
05:39 -!- vincenzopalazzo [~vincent@2001:b07:6474:9d49:849d:db24:7f93:fb8a] has joined #bitcoin-core-dev
05:40 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-dev
05:40 -!- owowo [~ovovo@unaffiliated/ovovo] has quit [Ping timeout: 240 seconds]
05:42 -!- molz_ [~mol@unaffiliated/molly] has quit [Ping timeout: 258 seconds]
05:44 -!- owowo [~ovovo@179.43.152.51] has joined #bitcoin-core-dev
05:44 -!- owowo [~ovovo@179.43.152.51] has quit [Changing host]
05:44 -!- owowo [~ovovo@unaffiliated/ovovo] has joined #bitcoin-core-dev
05:46 -!- da39a3ee5e6b4b0d [~da39a3ee5@2403:6200:8876:bbcd:f950:50c:999:2b9d] has quit [Ping timeout: 264 seconds]
05:49 -!- peterrizzo [~peterrizz@ool-44c18924.dyn.optonline.net] has quit [Quit: peterrizzo]
06:03 -!- peterrizzo_ [~peterrizz@ool-44c18924.dyn.optonline.net] has joined #bitcoin-core-dev
06:10 -!- Pavlenex [~Thunderbi@178.220.68.122] has joined #bitcoin-core-dev
06:24 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Ping timeout: 240 seconds]
06:34 -!- Pavlenex1 [~Thunderbi@185.244.212.67] has joined #bitcoin-core-dev
06:36 -!- davterra [~davterra@107.182.237.18] has joined #bitcoin-core-dev
06:37 -!- Pavlenex [~Thunderbi@178.220.68.122] has quit [Ping timeout: 264 seconds]
06:37 -!- Pavlenex1 is now known as Pavlenex
06:54 < wumpus> jonasschnelli: strange, you have a mismatch for the gitian macos signed build
06:55 < wumpus> but the unsigned one matches
07:00 -!- miketwenty1 [~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com] has joined #bitcoin-core-dev
07:07 < jonasschnelli> wumpus: that’s really strange. 
07:07 < jonasschnelli> Let me do it again
07:11 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev
07:14 -!- sr_gi [~sr_gi@80.174.218.168.dyn.user.ono.com] has quit [Read error: Connection reset by peer]
07:14 -!- sr_gi [~sr_gi@80.174.218.168.dyn.user.ono.com] has joined #bitcoin-core-dev
07:15 < wumpus> it shouldn't even be possible, a difference would invalidate the code-signing right?
07:16 < wumpus> or is there scope for malleability
07:18 -!- Pavlenex1 [~Thunderbi@141.98.103.251] has joined #bitcoin-core-dev
07:18 < jonasschnelli> The only reason I could think of is that I have built it before the sigs where pushed (then it would have took the rc2 detached sig).
07:19 < jonasschnelli> But I very much doubt that I did this
07:20 -!- Pavlenex [~Thunderbi@185.244.212.67] has quit [Ping timeout: 265 seconds]
07:20 -!- Pavlenex1 is now known as Pavlenex
07:22 < wumpus> it doesn't verify what it is attaching?
07:25 -!- Pavlenex [~Thunderbi@141.98.103.251] has quit [Quit: Pavlenex]
07:27 < jonasschnelli> I don’t think so. It just takes the newest signature from the 0.21 branch (signature repository)
07:28 < jonasschnelli> A check against the release/tag should probably be added.
07:28 < jonasschnelli> I investigate as soon as when I’m back on my desk
07:28 < wumpus> I mean a check that the signature checks out against what it's attached to
07:29 < wumpus> the windows one has a check like that IIRC
07:31 < jonasschnelli> wumpus: that’s not the case AFAIK. I don’t know if you can verify the signature on Linux. Probably possible but maybe complicated to add.
07:31 < wumpus> right, thanks
07:33 -!- Pavlenex [~Thunderbi@141.98.103.251] has joined #bitcoin-core-dev
07:38 -!- mol_ [~mol@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
07:46 -!- Pavlenex [~Thunderbi@141.98.103.251] has quit [Quit: Pavlenex]
07:48 < luke-jr> is the CI failing to merge with master before running? or did we lose a bunch of Cirrus instances?
07:49 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
07:49 < bitcoin-git> [bitcoin] ryanofsky closed pull request #19195: wallet: ScanForWalletTransactions cleanup (master...2020-06-06-scanforwallettransactions-cleanup) https://github.com/bitcoin/bitcoin/pull/19195
07:49 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
07:50 < luke-jr> also does this error make sense to anyone? only happens on bitcoinbuilds :/ https://bitcoinbuilds.org/index.php?ansilog=98434a62-d0b5-4596-bfde-5e5504998bc3.log
07:53 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 272 seconds]
07:54 < wumpus> luke-jr: it's passing two arguments to a function that takes one?
07:54 < wumpus> e.g. "CreateChainParams(gArgs, gArgs.GetChainName());"
07:54 -!- peterrizzo_ [~peterrizz@ool-44c18924.dyn.optonline.net] has quit [Read error: Connection reset by peer]
07:54 -!- peterrizzo_ [~peterrizz@ool-44c18924.dyn.optonline.net] has joined #bitcoin-core-dev
07:55 -!- mol [~mol@unaffiliated/molly] has joined #bitcoin-core-dev
07:55 -!- miketwen_ [~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com] has joined #bitcoin-core-dev
07:56 -!- reallll [~belcher@unaffiliated/belcher] has joined #bitcoin-core-dev
07:58 -!- miketwenty1 [~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com] has quit [Ping timeout: 240 seconds]
08:00 -!- reallll is now known as belcher
08:00 < wumpus> oh the two-arg version is correct in master, is this some rebase/merge problem maybe?
08:01 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has joined #bitcoin-core-dev
08:01 < wumpus> as somehow it ends up with an old chainparams.h with single-argument CreateChainParams
08:13 -!- Pavlenex [~Thunderbi@141.98.103.251] has joined #bitcoin-core-dev
08:22 -!- Pavlenex [~Thunderbi@141.98.103.251] has quit [Quit: Pavlenex]
08:25 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Remote host closed the connection]
08:27 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has joined #bitcoin-core-dev
08:34 -!- Bob_ [~bob@93.55.242.131] has joined #bitcoin-core-dev
08:35 < jonasschnelli> wumpus: I did the build again and getting now the “correct” hash.
08:35 < jonasschnelli> New build: https://bitcoin.jonasschnelli.ch/gitian/build/360
08:35 < jonasschnelli> Old build (the one I pushed to gitian.signs): https://bitcoin.jonasschnelli.ch/gitian/build/358
08:37 < wumpus> jonasschnelli: happy to hear that! time to upload the rc binaries then
08:37 < luke-jr> wumpus: I see. So that confirms a CI issue too >_<
08:37 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Quit: = ""]
08:37 < luke-jr> jonasschnelli: bitcoinbuilds is apparently not merging master before running CI
08:39 < jonasschnelli> luke-jr: AFAIK is does
08:40 < jonasschnelli> Someone, my old (invalid) build has different bitcoin-osx-unsigned.tar.gz as input
08:40 < jonasschnelli> I don't know why
08:40 < jonasschnelli> (old invalid build): https://bitcoin.jonasschnelli.ch/gitian/builds/360/bitcoin-dmg-signer-build.assert
08:41 < jonasschnelli> https://bitcoin.jonasschnelli.ch/gitian/builds/358/bitcoin-dmg-signer-build.assert
08:41 < jonasschnelli> very strange
08:42 < luke-jr> jonasschnelli: see https://bitcoinbuilds.org/index.php?ansilog=98434a62-d0b5-4596-bfde-5e5504998bc3.log
08:42 < jonasschnelli> I was just checking and came to the same conclusion luke-jr... hmm..
08:44 < luke-jr> jonasschnelli: if I fix my PR will it mess up your ability to troubleshoot?
08:44 < jonasschnelli> luke-jr: your right... so yes. bitcoinbuilds currently builds PR branches as they are (not merged on master)
08:44 < jonasschnelli> luke-jr: no. I'll know how to fix it.
08:44 < luke-jr> k
08:48 -!- Bob_ [~bob@93.55.242.131] has left #bitcoin-core-dev []
08:49 -!- Pavlenex [~Thunderbi@141.98.103.251] has joined #bitcoin-core-dev
08:49 -!- Pavlenex [~Thunderbi@141.98.103.251] has quit [Client Quit]
08:50 -!- kljasdfvv [~flack@p200300d46f24de007f9b1b51e45d0773.dip0.t-ipconnect.de] has quit [Quit: Konversation terminated!]
08:52 -!- jeremyrubin [~jr@c-73-15-215-148.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
08:54 -!- Barno7 [~bob@93.55.242.131] has joined #bitcoin-core-dev
08:55 -!- harrigan [~harrigan@ptr-93-89-242-235.ip.airwire.ie] has quit [Ping timeout: 256 seconds]
08:55 < sipa> wumpus: treating the block data as an import sounds like a good idea, and not too hard
09:02 < achow101> wumpus: yes, the prepared statement for each batch is so that we don't have issues where there are multiple batches for the same database. Although I'm pretty sure that we never have multiple batches writing to the same db at the same time anyeays.
09:03 < wumpus> sipa: great!
09:04 -!- mj_node [~mj_node@122.0.25.130] has joined #bitcoin-core-dev
09:04 < wumpus> achow101: right, thanks
09:06 -!- potato [~Thunderbi@240d:1a:3d4:7d00:6195:91a4:15a2:31e8] has joined #bitcoin-core-dev
09:11 -!- Pavlenex [~Thunderbi@141.98.103.251] has joined #bitcoin-core-dev
09:13 -!- alko89 [~alko89@unaffiliated/alko89] has quit [Quit: ZNC 1.7.5 - https://znc.in]
09:14 -!- alko89 [~alko89@unaffiliated/alko89] has joined #bitcoin-core-dev
09:28 -!- wumpus [~ircclient@pdpc/supporter/professional/wumpus] has quit [Quit: freebsd 12.2 update]
09:31 -!- Pavlenex [~Thunderbi@141.98.103.251] has quit [Quit: Pavlenex]
09:34 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has joined #bitcoin-core-dev
09:38 -!- wumpus [~ircclient@pdpc/supporter/professional/wumpus] has joined #bitcoin-core-dev
09:38 < wumpus> rc3 binaries up https://bitcoincore.org/bin/bitcoin-core-0.21.0/test.rc3/
09:57 < achow101> for some reason gverify is giving a mismatch for darosior's 0.20.2rc1 win and osx builds, but manual inspection does not show a difference
09:57 < luke-jr> O.o
10:00 < luke-jr> achow101: where do you see it?
10:00 < achow101> locally
10:01 < luke-jr> I also see no difference
10:01 < roconnor> does gverify look at filename dates or other file attributes?
10:02 < darosior> Hmm, gverify passes on my end
10:02 < luke-jr> achow101: could the signature be invalid/rejected for some reason?
10:02 < achow101> luke-jr: i don't think so
10:02 < darosior> Oh no, it does not
10:02 < darosior> Not for Windows, good catch..
10:03 < achow101> roconnor: iirc it expects the same filenames
10:03 < luke-jr> oh, found it
10:03 -!- samuel-pedraza [a5169307@gateway/web/cgi-irc/kiwiirc.com/ip.165.22.147.7] has joined #bitcoin-core-dev
10:03 < luke-jr> - release: v0.20.2rc1-win-unsigned
10:03 < luke-jr> I bet tha'ts it
10:04 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]
10:04 < achow101> ugh
10:04 < achow101> ok that doesn't matter
10:04 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-dev
10:05 < prusnak> Big Sur says Bitcoin Core installed from bitcoin-0.21.0rc3-osx.dmg is broken and should be moved to the bin; that said this happens on M1 system, so maybe on x86-64 it's fine
10:05 < prusnak> maybe jonasschnelli can confirm? ^
10:05 < achow101> prusnak: is it the notarization warning?
10:05 < prusnak> no, it's different than the error with rc2 (which i assume was notarization error)
10:07 < luke-jr> where did the dmg come from?
10:07 < luke-jr> could we have ended up with a signature that doesn't match a non-deterministic bin?
10:08 < dongcarl> Just tested on x86_64, same error
10:08 < luke-jr> :/
10:09 < luke-jr> what about the unsigned dmg?
10:09 < achow101> any more detail on that error?
10:10 < dongcarl> https://nextcloud.carl.homeserver.net/s/WXdFHLwwTygXZW4
10:11 < luke-jr> dongcarl: what is that and why does it demand I run JS?
10:12 < dongcarl> Direct link: https://nextcloud.carl.homeserver.net/s/WXdFHLwwTygXZW4/preview
10:12 < dongcarl> It's just the screenshot of the error when I open from the DMG (didn't drag to Applications)
10:12 < luke-jr> does that question mark on the bottom left tell anything more?
10:13 < achow101> hmm, damaged?
10:13 < sipa> damaged could mean the signature check fails, i guess
10:13 < dongcarl> https://nextcloud.carl.homeserver.net/s/7YTDtmrgaqPCYde/preview
10:14 < dongcarl> code signing went wrong
10:14 < luke-jr> sounds like it
10:14 < luke-jr> dongcarl: can you verify the unsigned DMG is okay?
10:14 < luke-jr> ping jonasschnelli 
10:14 < dongcarl> luke-jr: that's not uploaded yet I don't think
10:15 < dongcarl> Happy to test if anyone has one
10:15 < achow101> dongcarl: uploading it soon
10:17 < achow101> https://github.com/achow101/bitcoin/releases/tag/v0.21.0rc3
10:17 -!- samuel-pedraza [a5169307@gateway/web/cgi-irc/kiwiirc.com/ip.165.22.147.7] has quit [Quit: Connection closed]
10:18 < dongcarl> achow101: works
10:19 < prusnak> same here (on m1 system)
10:20 < luke-jr> sounds like jonasschnelli signed the wrong file, or committed the wrong sig
10:20 < luke-jr> or our combining process is suddnely broke
10:21 < achow101> i think he signed the wrong file
10:23 < roconnor> maybe I should keep my nose out of this, but what are the consequences of signing the wrong thing?  AFAIK publish signatures are unrevokable.
10:23 < roconnor> *published
10:23 < dongcarl> who does the windows codesigning?
10:23 < achow101> me
10:24 < dongcarl> roconnor: You might be thinking of notarization instead of codesigning?
10:24 < roconnor> maybe.  What's the difference?
10:24 < achow101> roconnor: in theory there shouldn't be any consequences because code signing is only done to shut up os warnings.
10:24 < roconnor> ah.
10:24 < achow101> but in practice, it both doesn't do that anymore, and it does convey some level of "this software is trusted" because users don't quite understand why we do that
10:25 < luke-jr> maybe we should just stop signing then?
10:25 < achow101> the consequence is that the software that is signed is malicious and some user is tricked into signing it
10:25 < achow101> *using it
10:25 < roconnor> maybe these code signatures are revokable?
10:26 < luke-jr> roconnor: these days, to *actually* shut macOS up, you have to opt in to Apple privacy violations (it tells them every time the app is opened, and which app it was)
10:26 < achow101> luke-jr: code signing is required for notarization, and notarization is now the thing to make macOS not give a warning
10:26 < luke-jr> achow101: but notarization is problematic, and we don't do it
10:26 -!- jonatack [~jon@213.152.186.40] has quit [Ping timeout: 240 seconds]
10:26 < luke-jr> so if the signature-alone is useless, why bother?
10:26 < achow101> iirc there's 2 levels of warnings, with signature alone being slightly less aggressive?
10:27 < achow101> and at least it still works on older macOS
10:27 < luke-jr> older macOS that we dropped support for? :P
10:27 < dongcarl> luke-jr: Without signing, the warning is much scarier... something like "unknown developer"
10:29 < achow101> I think there's an older macOS we support that doesn't do the notarization check
10:30 < luke-jr> anyway, jonas can prob fix this easily
10:30 -!- harrigan [~harrigan@ptr-93-89-242-235.ip.airwire.ie] has joined #bitcoin-core-dev
10:34 -!- kexkey [~kexkey@static-198-54-132-158.cust.tzulo.com] has joined #bitcoin-core-dev
10:45 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection]
10:46 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #bitcoin-core-dev
10:46 -!- troygiorshev [~troygiors@d67-193-140-136.home3.cgocable.net] has quit [Ping timeout: 260 seconds]
10:50 -!- jonatack [~jon@37.170.61.94] has joined #bitcoin-core-dev
10:58 -!- troygiorshev [~troygiors@d67-193-140-136.home3.cgocable.net] has joined #bitcoin-core-dev
11:03 -!- jonatack [~jon@37.170.61.94] has quit [Read error: Connection reset by peer]
11:08 < jonasschnelli> sorry.. was afk.
11:08 < jonasschnelli> Reading backlog
11:09 < jonasschnelli> I codesigned 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-0.21.0rc3-osx-unsigned.tar.gz
11:10 -!- chri_eb [~chris@gateway/tor-sasl/chrieb/x-28824719] has quit [Quit: chri_eb]
11:13 < jonasschnelli> achow101, luke-jr: I just checked the DMG is built initially and it works, signature is correct
11:13 < jonasschnelli> I got 998dddf3c0f9b568fc0c39e61e3d61d2843dfb968016b7ceaf23aca94ace2542  bitcoin-osx-signed.dmg as the only one
11:15 -!- yanmaani [~yanmaani@gateway/tor-sasl/yanmaani] has quit [Ping timeout: 240 seconds]
11:15 < jonasschnelli> But yes,... 46cfa036d365d69db2a3b78377621d6b214f2d78f3082f9c7ebd7a9b89cfc599  bitcoin-osx-signed.dmg has an invalid code signature
11:15 -!- yanmaani [~yanmaani@gateway/tor-sasl/yanmaani] has joined #bitcoin-core-dev
11:16 < jonasschnelli> but I signed 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-0.21.0rc3-osx-unsigned.tar.gz
11:16 < jonasschnelli> let me try agai
11:17 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #bitcoin-core-dev
11:20 < jonasschnelli> I double checked and I can confirm that I have signed 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-0.21.0rc3-osx-unsigned.tar.gz
11:20 < achow101> is signing deterministic?
11:20 < jonasschnelli> The signature is not deterministic,... doing it again gives me a different file/hash
11:20 < jonasschnelli> I can't verify what went wrong
11:22 < jonasschnelli> I have verified the signed 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-0.21.0rc3-osx-unsigned.tar.gz and it works
11:22 < jonasschnelli> I don't know what to do,... I can sign and push again. But I rather know why it happend
11:23 < achow101> How can you check what was signed?
11:23 < jonasschnelli> I keep the files
11:24 < achow101> ah
11:25 < achow101> I guess for an rc it's fine to leave it, but in the future we should test the signed binary first
11:26 -!- jonatack [~jon@213.152.161.244] has joined #bitcoin-core-dev
11:26 < wumpus> we'll correct it for the next rc (or final) I guess, it's too bad the sig verification can't run in linux
11:26 < achow101> we can't verify the signature after signing?
11:27 < sipa> we have a linux based signing tool, right?
11:28 < achow101> only for windows afaik
11:28 < sipa> i'd be surprised if it doesn't have most of the code needed for verification too
11:28 < sipa> oh
11:28 < achow101> for windows we can (and do I think) verify the sig after signing
11:30 < sipa> so the osx signature is created on osx, but stapled onto the binary in gitian/linux ?
11:30 < sipa> *macos
11:31 < achow101> yes
11:33 < luke-jr> jonasschnelli: can you upload or compare 998dddf3c0f9b568fc0c39e61e3d61d2843dfb968016b7ceaf23aca94ace2542 with 46cfa036d365d69db2a3b78377621d6b214f2d78f3082f9c7ebd7a9b89cfc599 ?
11:33 < luke-jr> I guess our combiner is malfunctioning?
11:33 -!- luke [~luke@bitnomial/staff/luke] has joined #bitcoin-core-dev
11:34 < jonasschnelli> luke-jr: 46cfa036d365d69db2a3b78377621d6b214f2d78f3082f9c7ebd7a9b89cfc599  is here -> https://bitcoin.jonasschnelli.ch/gitian/build/360
11:39 < jonasschnelli> luke-jr: I don't have 46cfa036d365d69db2a3b78377621d6b214f2d78f3082f9c7ebd7a9b89cfc599
11:39 < jonasschnelli> my first gitian build went somehow wrong: https://bitcoin.jonasschnelli.ch/gitian/build/358
11:40 < jonasschnelli> the build log from the signer was actually a full macOS unsigned build: https://bitcoin.jonasschnelli.ch/gitian/builds/358/build_osx_signer.log (for whatever reason)
11:41 < jonasschnelli> I think the dmg hash mismatch has nothing to do with the invalid signature
11:46 < jonasschnelli> hmm...
11:47 < jonasschnelli> I think what I need to do before pushing the macOS signatures is doing a gbuild with gitian-descriptors/gitian-osx-signer.yml with the detached signature
11:47 < jonasschnelli> as sort of a dbl-check
11:51 < luke-jr> would still be nice to figure out why it didn't work this time :x
11:53 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
11:53 < bitcoin-git> [bitcoin] vova557 opened pull request #20628: Владелец (master...patch-10) https://github.com/bitcoin/bitcoin/pull/20628
11:53 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
11:54 -!- greypw [~greypw@unaffiliated/greypw] has quit [Quit: I'll be back!]
11:54 < jonasschnelli> luke-jr: Indeed. I'm still trying to figure that out.
11:54 -!- greypw [~greypw@unaffiliated/greypw] has joined #bitcoin-core-dev
12:02 -!- davec [~davec@072-183-054-196.res.spectrum.com] has quit [Quit: leaving]
12:03 -!- miketwen_ [~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com] has quit [Ping timeout: 240 seconds]
12:11 < jonasschnelli> one of the main problem is, that I currently can't test the detached signature before pushing it to the git repository
12:11 < jonasschnelli> I try now to modify the gitian descriptor to allow that
12:12 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
12:12 < bitcoin-git> [bitcoin] jonasschnelli closed pull request #20628: Владелец (master...patch-10) https://github.com/bitcoin/bitcoin/pull/20628
12:12 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
12:13 < achow101> jonasschnelli: you can put it in gitian-builder/inputs/signature
12:13 < achow101> and make a local tag
12:13 < jonasschnelli> and set --commit signature=<tag> to that custom tag?
12:14 < achow101> i just set it to the actual version name
12:14 < achow101> and delete it later for the actual build
12:15 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Ping timeout: 240 seconds]
12:15 < jonasschnelli> but AFAIK the signer always pulls https://github.com/bitcoin-core/bitcoin-detached-sigs.git?
12:15 < achow101> it pulls but it shouldn't do anything
12:15 < jonasschnelli> and overwrites the git?
12:15 < achow101> it doesn't overwrite
12:15 < jonasschnelli> ah.
12:16 < jonasschnelli> I added now a part to the descriptor that allows putting a signatures.tar.gz in inputs
12:16 < jonasschnelli> and if there,... it'll take it
12:17 < luke-jr> jonasschnelli: -u bitcoin=/path/to/local/bitcoin
12:18 < luke-jr> tells gbuild where to pull from
12:18 < luke-jr> I always have ti pull from local
12:18 < jonasschnelli> luke-jr: https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-descriptors/gitian-osx-signer.yml#L11
12:18 < jonasschnelli> I guess it always pulls it from there.
12:19 < midnight> Oh nice, no changes necessary in the gitian signing env and I'm getting matches.
12:19 < luke-jr> not if you specify -u
12:19 < luke-jr>     -u, --url PAIRS                  comma separated list of DIRECTORY=URL pairs
12:19 < luke-jr> not very useful --help lol
12:19 < jonasschnelli> I do that for bitcoin=
12:19 < jonasschnelli> but wasn't aware you can also do it then for signature=
12:20 < luke-jr> it's no different ;)
12:20 < jonasschnelli> However,.. I'm going for the additional input file
12:20 < midnight> just me and wumpus I think with the windows sigs for the rc but
12:20 < jonasschnelli> Seems dummysave
12:20  * midnight shrug
12:21 < luke-jr> midnight: ?
12:21 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has joined #bitcoin-core-dev
12:21 < midnight> luke-jr: working nicely still, in spite of my semi-custom build environment.
12:21 < luke-jr> midnight: I see lots of windows sigs for 0.21.0rc3
12:21 < midnight> hrm
12:24 < jonasschnelli> midnight: I guess semi-custom is good. Better than if everyone uses the same host/env with the same script
12:24 < luke-jr> my attempt to semi-custom was met with mismatching
12:24 < luke-jr> but my idea of semi-custom was a ppc64le VM image :P
12:24 < jonasschnelli> have you figured out why?
12:25 < midnight> jonasschnelli: The maintenance of the custom(ish) build env is entirely on me, and I figure it's better to arrive at similar results this way.
12:25 < luke-jr> jonasschnelli: apparently the ppc64le compilers do not output the same objects as the x86_64 compilers, even cross compiling
12:25 < luke-jr> no idea why :/
12:25 < midnight> triangulation and all that.
12:26 < luke-jr> you would *think* i686-w64-mingw64-gcc (or whatever) would produce the same stuff on x86_64 and ppc64le, but apparently not
12:28 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 272 seconds]
12:30 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
12:30 < bitcoin-git> [bitcoin] dongcarl opened pull request #20629:  depends: Improve id string robustness  (master...2020-12-improve-depends-id-string) https://github.com/bitcoin/bitcoin/pull/20629
12:30 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
12:32 < jonasschnelli> i'm doing the 0.20.2 mac signatures asap (need to track-down the previous issue first)
12:44 -!- belcher [~belcher@unaffiliated/belcher] has joined #bitcoin-core-dev
12:46 < jonasschnelli> is there a way to speeup gitians "Upgrading system, may take a while (log in var/install.log)"?
12:46 < jonasschnelli> Re-create the base system?
12:46 < jonasschnelli> s/system/image
12:47 < sipa> iirc, yes
12:47 < sipa> creating an image after the update speeds things up
12:48 < jonasschnelli> the install part takes >10mins here.. annoying for testing
12:48 < wumpus> yes, the "Upgrading image" takes longer than the actual builds here
12:48 < jonasschnelli> ideed
12:49 < jonasschnelli> +n
12:49 < wumpus> I have no idea how to speed it up, I hacked it once to upgrade the base image but then somehow ended up with two versions of every package and non-deterministic results, so yea, if regenerating the base image solves it I'd definitely recommend that path
12:50 < jonasschnelli> Okay... I'll try that
12:50 < wumpus> fairly sure that works for one of {KVM, LXC}, don't know which one, the other always gets the ancient image
12:51 < wumpus> (they both generate the base image in completely different ways)
12:53 < wumpus> it would make sense to look into it some day but also guix builds are around the corner right
12:54 < sipa> Any Day Now(tm)
12:55 < wumpus> releases are sufficiently rare anyway, but the extra delays are mostly annoying when testing/iterating something
13:04 -!- tylerchambers [uid477594@gateway/web/irccloud.com/x-raafweykxwvatycu] has joined #bitcoin-core-dev
13:10 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Ping timeout: 240 seconds]
13:11 < jonasschnelli> I'm really confused
13:12 < jonasschnelli> I have downloaded again 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11  bitcoin-osx-unsigned.tar.gz ...
13:12 < jonasschnelli> ./detached-sig-create.sh -s "Bitcoin"
13:12 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has joined #bitcoin-core-dev
13:12 < jonasschnelli> verified code-signature  codesign -v dist/Bitcoin-Qt.app
13:13 < jonasschnelli> uploaded signature.tar.gz
13:13 < jonasschnelli> did a gitian signer build
13:13 < jonasschnelli> downloaded the dmg
13:13 -!- luke [~luke@bitnomial/staff/luke] has quit [Quit: sleep]
13:13 < jonasschnelli> verified the signature and I get:
13:13 < jonasschnelli> * /Volumes/Bitcoin-Core 4/Bitcoin-Qt.app: invalid signature (code or signature have been modified)
13:13 < achow101> huh. did apple change something recently?
13:14 < sipa> jonasschnelli: this is an old version?
13:14 < sipa> of bitcoin core?
13:14 < jonasschnelli> sipa: no.. still trying to figure out what happend for 0.21.0rc3
13:15 < jonasschnelli> AFAIK 5e3a08ae8195190d6f1b12e3e1e9d710e7ad385941a6e8d04e3391f12deddb11 was the unsigned tarball (bitcoin-osx-unsigned.tar.gz)
13:16 < achow101> yep
13:16 < jonasschnelli> I redid everything and still getting invalid signatures
13:16 < jonasschnelli> Might it be possible that there is an edge-case with the signature size or the attaching of the signature?
13:20 < achow101> No issues on 10.15.7, so maybe just a big sur problem
13:21 < achow101> nvm, didn't click far enough
13:22 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
13:22 < bitcoin-git> [bitcoin] jonasschnelli opened pull request #20630: Allow providing local signatures in gitian osx signer (master...2020/12/macos_gitian_signer) https://github.com/bitcoin/bitcoin/pull/20630
13:22 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
13:25 < jonasschnelli> Here is the gist (how I did it): https://gist.github.com/jonasschnelli/39f0b03981e1bbffff929f538cf139a9
13:25 < jonasschnelli> I have absolutely no clue why the signatures are wrong
13:28 < jonasschnelli> if anyone like to go down the rabbit hole:
13:28 < jonasschnelli> my local signed and detached signature directory: https://bitcoin.jonasschnelli.ch/bitcoin-osx-unsigned-nowsigned.zip
13:28 < jonasschnelli> ended up in build: https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed3.dmg
13:28 < jonasschnelli> (both files are in the gist above)
13:29 < jonasschnelli> maybe cfields has some insights?
13:30 < jonasschnelli> I need some sleep and try it again with the 0.20.2rc1 release (and see what happens there)
13:30  * jonasschnelli afk
13:34 -!- Guyver2 [Guyver@guyver2.xs4all.nl] has quit [Quit: Going offline, see ya! (www.adiirc.com)]
13:36 -!- vincenzopalazzo [~vincent@2001:b07:6474:9d49:849d:db24:7f93:fb8a] has quit [Quit: Leaving]
13:48 -!- eugene-ff [~eugene_ff@2604:2000:1383:472b:759a:a9e9:8a19:69d2] has joined #bitcoin-core-dev
14:00 < midnight> gah, xcode is a 7+ GB download.
14:00 < sipa> yes :(
14:03 < sipa> and we only need 47 MB from it
14:11 -!- Talkless [~Talkless@mail.dargis.net] has joined #bitcoin-core-dev
14:29 < achow101> interesting. there's a single byte difference between jonasschnelli signing result and his gitian result
14:34 < sipa> what byte?
14:35 < achow101> https://0bin.net/paste/NuOSAQ26#QxBJq7yNS7ynp6oMsXzvKQWT0vXTLYboU2QRfyBqu2d
14:36 < sipa> if you fix it, does it verify?
14:36 < achow101> yes, the original verifies
14:38 < sipa> what do you mean by signing result and gitian result?
14:40 < sipa> the output of the macos signing tool, and the result of gitian applying the extracted signature?
14:40 < achow101> the results from https://bitcoin.jonasschnelli.ch/bitcoin-osx-unsigned-nowsigned.zip which come from the signing process. it contains a fully signed binary
14:40 < sipa> got it
14:40 < achow101> the gitian result is https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed3.dmg which comes from gitian builder combining the unsigned with the signature.tar.gz from the above zip
14:40 < sipa> so the extraction/reattaching process doesn't result in the exact same binary
14:41 < achow101> yeah
14:43 < sipa> can you run "pagestuff -p" on both?
14:45 < achow101> identical
14:47 < sipa> so
14:48 < sipa> the detached-sig-create.sh tool uses pagestuff -p | tail -2 | grep offset | sed 's/[^0-9]*//g' to figure out the offset
14:49 < sipa> and the size of the sign file for the size
14:49 < achow101> the sizes are off by one byte
14:49 < sipa> aha
14:49 < sipa> that's what i was going to ask next
14:49 < sipa> i think this is the cause
14:50 < achow101> x86_64-apple-darwin-otool -l Bitcoin-Qt for segment "__LINKEDIT" says "vmsize 0x000000000007b000" but on the other it's "vmsize 0x000000000007c000"
14:50 < sipa> which one is bigger?
14:50 < achow101> gitian one is bigger
14:50 < achow101> wait no, that's backwards
14:50 < achow101> signed one is bigger, gitian is smaller
14:52 < sipa> that's a 4 kB difference though
14:52 < sipa> not 1 byte
14:53 < achow101> The files are the same size
14:53 < achow101> and diffoscope only sees the difference on that single byte
14:54 < sipa> what if you run the same on the unsigned gitian output (= the input to the macos signing tool)
14:54 < achow101> not sure what you mean
14:55 < sipa> x86_64-apple-darwin-otool -l Bitcoin-Qt
14:55 < sipa> oh
14:55 < sipa> nvm
14:56 < sipa> yeah, what if you run that on the unsigned Bitcoin-Qt binary?
14:57 < achow101> __LINKEDIT segment is the same size as the gitian result
14:58 < sipa> so
14:59 < sipa> the codesigning tool modified the __LINKEDIT segment, but applying the detached signature didn't do the same?
14:59 < achow101> seems so
15:01 < sipa> if you run "pagestuff -p Bitcoin-Qt | tail -2 | grep size | sed 's/[^0-9]*//g'", what do you get?
15:02 < sipa> on the various version
15:06 < achow101> ah wait, I was looking at the wrong file earlier
15:06 < sipa> where do the linux binaries pagestuff and codesign_allocate come from?
15:07 < achow101> the otool on the unsigned binary gives a way smaller __LINKEDIT, presumably because the code signature is involved in there
15:07 < sipa> i guess that's expected
15:08 < achow101> they come from one of the depends packages, not sure which
15:08 < sipa> ah i see
15:11 < achow101> the sizes from that pagestuff command are the same for both gitian and signed
15:11 < sipa> and the offsets too?
15:11 < achow101> not the same for unsigned, but that's expected because the section size it looks for doesn't exist as it's the code sig
15:11 < sipa> yeah
15:11 < achow101> same offests
15:14 < sipa> ok, so the byte that changed is in the MP_MACH_HEADERS section, in the very beginning of the file
15:14 < sipa> while the codesig is at the very end
15:15 < achow101> presumably that section is a table
15:21 < achow101> so the problem must be with codesign_allocate because that's what sets that size value
15:22 -!- davterra [~davterra@107.182.237.18] has quit [Quit: Leaving]
15:25 < sipa> the codesign tool has a --detached option to construct detached signatures directly
15:26 < sipa> why is the tooling doing signing directly, and then extracting the signatures from the result?
15:26 < sipa> this may be a reason for a difference
15:26 < achow101> this workflow probably was setup before that existed
15:27 < sipa> is it possible that the codesign_allocate tool is out of date?
15:28 < achow101> possibly
15:29 < sipa> what else does otool or anything else say about __LINKEDIT?
15:32 < achow101> nothing else afaict
15:35 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 272 seconds]
15:38 < sipa> achow101: does macos come with a codesign_allocate tool too?
15:39 -!- AmberJ_ [~AmberJ_@178.239.168.171] has quit [Remote host closed the connection]
15:40 < achow101> yes
15:40 < sipa> the native_cctools depends package is only a few months old, and most changes since have been for apple ARM stuff it seems
15:43 < sipa> achow101: if you run "codesign_allocate -i <unsigned Bitcoin-Qt filename> -a x86_64 225312 -o <some temp file>
15:43 < sipa> on macos
15:44 < sipa> and then inspect that temp file, does it have the correct __LINKEDIT vmsize?
15:44 < achow101> will try
15:46 -!- peterrizzo_ [~peterrizz@ool-44c18924.dyn.optonline.net] has quit [Quit: peterrizzo_]
15:48 -!- mol [~mol@unaffiliated/molly] has joined #bitcoin-core-dev
15:48 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has quit [Ping timeout: 258 seconds]
15:52 < achow101> It's the correct size
15:53 < sipa> hmm, we should try updating to the latest native_cctools package i guess and see if that fixes it
15:57 < sipa> an alternative is making jonasschnelli downgrade his codesign_allocate tool to one that's compatible with the native_ccttols we use, and then tell his codesign tool to use that
15:59 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]
15:59 < achow101> I wonder if we're just running into some weird edge case now
16:00 -!- r2wj [~r2wj@185.204.1.185] has joined #bitcoin-core-dev
16:02 < achow101> I'll try an updated native_cctools first
16:02 < sipa> cool
16:03 < sipa> it's also possible that jonasschnelli's codesign tool is too old!
16:03 < sipa> (as native_cctools was updated for 0.21)
16:03 < achow101> Well it's worked for the previous rc
16:04 < sipa> it may be that some alignment thing changed that doesn't matter in every situation
16:06 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Remote host closed the connection]
16:07 < sipa> maybe we should just make some arbitrary binary-changing commits for rc4, and hope that that fixes it
16:07 < sipa> *hides*
16:09 -!- Talkless [~Talkless@mail.dargis.net] has quit [Quit: Konversation terminated!]
16:09 -!- mol_ [~mol@unaffiliated/molly] has joined #bitcoin-core-dev
16:12 < sipa> jonasschnelli: what version is your cctools? if you have the ability to change it, use 949.0.1 
16:13 < sipa> achow101: what version is your codesigning tool on macos?
16:13 < achow101> no idea
16:13 < achow101> there's no version command for codesign or codesign_allocate
16:14 -!- mol [~mol@unaffiliated/molly] has quit [Ping timeout: 265 seconds]
16:14 < sipa> this line in codesign_allocate changed in the latest cctools:
16:15 < sipa> object->seg_linkedit->vmsize = rnd32(object->seg_linkedit->filesize, get_segalign_from_flag(&arch_signs[i].arch_flag));
16:15 < achow101> where do you see this?
16:17 < sipa> https://github.com/tpoechtrager/cctools-port/commit/466063c7f7486762a5a8f0d98c48dc5ffbe42f74#diff-c0274ccf413d9c63c15dc20f765faa1dd750ae7a74b65dfda9afbe1e91ea137cR711
16:17 < sipa> this is in the version we're using in the gitian build
16:17 < sipa> but given that it recently changed (may 2020), perhaps somehow jonasschnelli is using an apple equivalent that is older
16:17 < sipa> *march 2020
16:19 < sipa> actually, no
16:19 < sipa> rnd -> rnd32 doesn't change anything
16:21 < achow101> eh.. they don't make tags for new versions.. great
16:22 -!- dermoth_ [~dermoth@unaffiliated/dermoth] has joined #bitcoin-core-dev
16:23 -!- dermoth [~dermoth@unaffiliated/dermoth] has quit [Disconnected by services]
16:23 -!- dermoth_ is now known as dermoth
16:24 < sipa> got it!
16:24 < sipa> cctools v921 uses 4k as default alignment for x86_64
16:24 < sipa> oh, no, i misread :(
16:28 < sipa> what is the full output from that x86_64-apple-darwin-otool -l Bitcoin-Qt command?
16:28 -!- proofofkeags [~proofofke@174-16-212-53.hlrn.qwest.net] has joined #bitcoin-core-dev
16:29 < achow101> of which version of Bitcoin-Qt
16:30 < sipa> no matter
16:31 < achow101> https://0bin.net/paste/ArdAXxJ3#67qz6LxpqPGqIXWODToWwyFxorHcouHH1CzJGxdBB2I
16:31 < gribble> https://github.com/bitcoin/bitcoin/issues/67 | does not compile with boost_1_45_0 on OS X 10.6 · Issue #67 · bitcoin/bitcoin · GitHub
16:31 < achow101> bad bot
16:35 < sipa> the __LINKEDIT "filesize" entry is the same for the codesign-output-binary and the gitian-reconstructed one, right?
16:35 < sipa> just vmsize is different
16:35 < achow101> yes
16:36 < sipa> yeah, if it wasnt't, more than 1 byte would differ
16:38 < sipa> literally every version of cctools i can find uses 0x1000 as segment alignment for X86_64
16:38 < sipa> so the correct rounding of filesize 500144 to vmsize is 0x7b000
16:39 < sipa> why is jonas' tool rounding it to 0x7c000 (like he's using a segment alignment of 0x2000 or 0x4000)
16:39 < sipa> now, libstuff uses 0x2000 for unknown CPU architectures
16:40 < sipa> so maybe somehow the wrong architecture is passed down?
16:41 < sipa> see https://opensource.apple.com/source/cctools/cctools-949.0.1/libstuff/arch.c.auto.html look for "{ CPU_TYPE_X86_64,    LITTLE_ENDIAN_BYTE_SEX, 0x7fff5fc00000LL,  0x1000 },"
16:41 < achow101> updating cctools (specifically codesign_allocate) did not make a difference
16:42 -!- jeremyrubin [~jr@c-73-15-215-148.hsd1.ca.comcast.net] has quit [Ping timeout: 260 seconds]
16:42 < achow101> jonasschnelli: are you using the same signing machine as before?
16:43 < sipa> and in https://opensource.apple.com/source/cctools/cctools-949.0.1/misc/codesign_allocate.c.auto.html look for "object->seg_linkedit->vmsize ="
16:43 < achow101> IIRC he has one of the M1 macs for testing stuff, maybe he used that for signing?
16:44 -!- Eagle[TM] [~EagleTM@unaffiliated/eagletm] has quit [Ping timeout: 240 seconds]
16:50 < sipa> i suspect that if we add 1024 bytes to the sig size in codesign_allocate it may work
16:50 < sipa> if you do:
16:51 < sipa> codesign_allocate -i <unsigned Bitcoin-Qt filename> -a x86_64 $((225312+1024)) -o <some temp file>
16:51 < sipa> on linux
16:51 < sipa> it may just work, with the existing signature
16:51 < achow101> I'm wondering why this size makes a difference to how the signature validates
16:51 < achow101> istm it shouldn't be covered by the signature itself
16:52 < sipa> i believe the code signature covers the entire binary, with the signature itself wiped out (this is a guess...)
16:52 < sipa> so its size matters
16:52 < sipa> that would explain why a tool like codesign_allocate exists in the first place
16:52 < sipa> as its behavior is nominal for the signing process
16:52 < achow101> but the size of the signature isn't known until the signature is made
16:53 < sipa> i assume that the codesign tool does now the signature size beforehand
16:53 < sipa> *know
16:53 < sipa> or it pads it to make sure it does
16:53 < achow101> i suppose it just needs to fall within the page size
16:53 < achow101> and that's probably a big enough range
16:53 < sipa> right
16:54 < achow101> so apple makes codesign_allocate open source, but not codesign itself?
16:54 < sipa> apparently
16:54 < sipa> but codesign invokes codesign_allocate
16:55 < achow101> supposedly
16:55 < sipa> and you can override which codesign_allocate to use... as long as it's signed by apple
16:56 < sipa> if you run "codesign_allocate -i <gitian Bitcoin-Qt filename> -a x86_64 $((225312+1024)) -o <newname>" in linux, does the resulting binary work in macos?
16:57 < sipa> oh, nvm
16:57 < sipa> nope, won't work
16:57 < sipa> that'll actually make space in the binary
16:58 -!- chjj_ [~chjj@unaffiliated/chjj] has joined #bitcoin-core-dev
17:00 < achow101> what if I set arch to ARM64 so it uses 0x4000 as the page size?
17:00 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-dev
17:03 < achow101> and codesign_allocate doesn't like it
17:08 -!- chjj [~chjj@unaffiliated/chjj] has quit [Quit: WeeChat 2.2]
17:08 < sipa> yeah i don't expect that'd work
17:08 < sipa> achow101: but if you run codesign_allocate with 225312, on macos, you also get the 0x7c000 vmsize?
17:09 -!- chjj_ [~chjj@unaffiliated/chjj] has quit [Quit: WeeChat 2.9]
17:10 -!- chjj [~chjj@unaffiliated/chjj] has joined #bitcoin-core-dev
17:11 < achow101> Ah, yes
17:12 < sipa> so perhaps... the codesign tool isn't invoking it with size 225312
17:12 < sipa> mayne it invokes it twice
17:13 < sipa> once with a prospective size, to get the to-be-signed binary, and then again with the real size
17:13 < achow101> Now, I think the installation of codesign_allocate is different from the open source one
17:13 < sipa> that'd explain the filesize of LINKEDIT being correct, but vmsize isn't
17:14 < achow101> Because it matches jonas's, not gitian
17:17 -!- mj_node [~mj_node@122.0.25.130] has quit [Quit: Leaving]
17:19 < sipa> right
17:23 < sipa> if you on linux run codesign_allocate with a larger size first, and then again withnthe correct size, does the __linkedit stuff end up being correct?
17:33 < achow101> what's a big enough size?
17:35 < sipa> $((225312+1024))
17:39 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]
17:39 < achow101> nope. also still 7b
17:40 < sipa> after both commands, or even after the first one?
17:40 < achow101> both
17:40 < sipa> oh, i'm wrong
17:40 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #bitcoin-core-dev
17:40 < achow101> well both as in both both commands and first one
17:40 < sipa> 0x1000 is not 1024, but 4096
17:41 < sipa> so $((225312+4024))
17:41 < sipa> so $((225312+4096))
17:41 < achow101> oh right
17:43 < achow101> that works
17:43 < achow101> although I will say that this is not a suitable workaround
17:44 < sipa> we could produce in the gitian output the information on what size to resize to
17:45 < sipa> it can be deterministically inferred from the macos-signed binary
17:46 < achow101> i suppose
17:46 < achow101> it feels like there should be a better way
17:47 < sipa> yes
17:47 < sipa> there is clearly a discrepancy between the macos codesign_allocate tool and the opensource one
17:47 < sipa> and i'm confused why that isn't a problem for other projects
17:48 < achow101> maybe no one has tried to make a new release recently?
17:49 < achow101> we didn't see this a few weeks ago for rc2
17:52 < sipa> yeah, baffling
17:54 < sipa> lol @ actively {-beating our heads against Apple's walled garden wall-} debugging this issue
17:56 < sipa> there is a macports version of cctools, presumably using the open source code
17:56 < sipa> if you run that on macos, what behavior does it have?
17:57 < achow101> I don't have macports installed
18:01 < achow101> I found an old version of codesign source on apple's opensource site: https://opensource.apple.com/source/security_systemkeychain/security_systemkeychain-55191.1/src/codesign.cpp.auto.html
18:01 < achow101> it disappears eventually, but that might just mean it got moved to somewhere else
18:02 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit []
18:02 < sipa> that one doesn't invoke codesign_allocate
18:02 < sipa> as far i can see, but it may be in another source file
18:12 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has joined #bitcoin-core-dev
18:18 -!- ConsciousFlesh [~cFlesh@unaffiliated/consciousflesh] has joined #bitcoin-core-dev
18:21 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has joined #bitcoin-core-dev
18:22 -!- twistedline [~twisted@unaffiliated/twistedline] has quit [Read error: Connection reset by peer]
18:24 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has quit [Remote host closed the connection]
18:24 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has joined #bitcoin-core-dev
18:27 < achow101> sipa: macports cctools uses 0x7b000
18:31 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has quit [Remote host closed the connection]
18:31 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has joined #bitcoin-core-dev
18:32 < ConsciousFlesh> Are there any devs here who were involved with, or around when the `RPCHelpMan()` function replaced throwing a runtime error to display JSON-RPC help text?
18:33 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has quit [Remote host closed the connection]
18:33 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has joined #bitcoin-core-dev
18:33 < sipa> achow101: great, so the apple codesign_allocate just behaves differently from their published source code
18:48 -!- ConsciousFlesh [~cFlesh@unaffiliated/consciousflesh] has left #bitcoin-core-dev ["If anyone was involved in the RPCHelpMan update I mentioned could you pm me? Had some questions about it's implementation."]
18:49 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has quit [Remote host closed the connection]
18:49 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has joined #bitcoin-core-dev
18:54 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has quit [Remote host closed the connection]
18:55 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has joined #bitcoin-core-dev
18:58 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has quit [Ping timeout: 246 seconds]
18:59 < achow101> i guess we shouldn't be that surprised that they do different things
19:24 -!- twistedline_ [~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8] has quit [Remote host closed the connection]
19:26 -!- miketwenty1 [~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com] has joined #bitcoin-core-dev
19:32 -!- twistedline [~twisted@unaffiliated/twistedline] has joined #bitcoin-core-dev
19:52 < achow101> sipa: adding the run codesign_allocate twice trick to detached-sig-apply.sh does make rc3 pass signature validation
19:52 < sipa> great
19:53 < sipa> so we just need something to take a size from the code-signatures.tgz
19:53 < sipa> and modify jonas' script to compute what the right pre-allocate size is
19:53 < achow101> I wonder if maybe it would be easier to codesign_allocate the unsigned dmg with a larger than necessary but fixed sized chunk
19:54 < achow101> the gitian signer's codesign allocate shrinks it, but I don't think effects the vmsize
19:54 < sipa> looking at the codesign_allocate tool, it just only ever increments the vmsize
19:55 < sipa> which is going to be the rounded-up version of the largest value the filesize ever had
19:55 < sipa> and whatever we do, we need to replicate the apple tool's behavior
19:55 < achow101> right, so we could always overshoot by a lot when we make the unsigned dmg
19:55 < sipa> yup
19:55 < sipa> that's a possibility
19:56 < luke-jr> [20:46:44] <jonasschnelli> is there a way to speeup gitians "Upgrading system, may take a while (log in var/install.log)"? <-- I just run the base VM and update it
19:56 < achow101> I'm going to go through history and see what the largest sigs we've had before is
19:57 < achow101> any idea if an overly large vmsize has any negative effects?
19:59 < sipa> i suspect if we crank up the codesign --pagesize value the sig will be much smaller
20:00 < sipa> it signs every page separately
20:01 < achow101> oh, is that why the sig is so big and varies a lot
20:12 -!- davec [~davec@072-183-054-196.res.spectrum.com] has joined #bitcoin-core-dev
20:43 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #bitcoin-core-dev
20:56 -!- mj_node [~mj_node@122.0.25.130] has joined #bitcoin-core-dev
21:00 -!- Emcy [~Emcy@unaffiliated/emcy] has quit [Read error: Connection reset by peer]
21:01 -!- Emcy [~Emcy@unaffiliated/emcy] has joined #bitcoin-core-dev
21:05 -!- bitcoin-git [~bitcoin-g@x0f.org] has joined #bitcoin-core-dev
21:05 < bitcoin-git> [bitcoin] pox opened pull request #20635: fix misleading comment about call to non-existing function (master...master) https://github.com/bitcoin/bitcoin/pull/20635
21:05 -!- bitcoin-git [~bitcoin-g@x0f.org] has left #bitcoin-core-dev []
21:08 -!- dhruvm [~noreply@c-73-158-59-66.hsd1.ca.comcast.net] has quit [Ping timeout: 240 seconds]
21:17 -!- r2wj [~r2wj@185.204.1.185] has quit [Remote host closed the connection]
21:18 -!- fearbeag [~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca] has joined #bitcoin-core-dev
21:23 -!- flag [~flag@net-93-66-71-105.cust.vodafonedsl.it] has quit [Quit: leaving]
21:25 -!- sledges [~sledges@185.163.110.125] has joined #bitcoin-core-dev
21:31 -!- flag [~flag@net-93-66-71-105.cust.vodafonedsl.it] has joined #bitcoin-core-dev
21:33 -!- fearbeag [~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca] has quit [Ping timeout: 260 seconds]
22:22 -!- miketwenty1 [~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com] has quit [Remote host closed the connection]
22:44 -!- mj_node [~mj_node@122.0.25.130] has quit [Remote host closed the connection]
22:44 -!- mj_node [~mj_node@122.0.25.130] has joined #bitcoin-core-dev
23:00 < jonasschnelli> sipa, achow101: I'm reading the backlog... for your info,... I haven't changed anything on my signing computer since signing rc2 (which was successful).
23:01 < achow101> jonasschnelli: good to know. the backlog is quite a journey..
23:01 < jonasschnelli> AFAIK detached-sig-create.sh under the hood, it uses the macOS codesign tool (which I haven't changed)
23:01 < jonasschnelli> achow101: yeah.. halfway through.
23:01 < jonasschnelli> Very interesting.
23:03 < jonasschnelli> 2020-12-11T23:57:47  <sipa> an alternative is making jonasschnelli downgrade his codesign_allocate tool to one that's compatible with the native_ccttols we use, and then tell his codesign tool to use that
23:03 < jonasschnelli> ^ as said,... I haven't upgraded anything since rc2
23:04 < jonasschnelli> 2020-12-12T00:12:49  <sipa> jonasschnelli: what version is your cctools? if you have the ability to change it, use 949.0.1
23:05 < jonasschnelli> I guess sipa wants to know what clang version I have on my signing mac. Why would this be relevant? IMO codesign is not part of the llvm suite
23:05 < jonasschnelli> And cctools on the gitian host should be irrelevant
23:05 < achow101> cctools is separate from clang
23:05 < achow101> it's part of xcode's developer tools
23:06 < achow101> jonasschnelli: the tl;dr is that codesign_allocate as distributed by apple is different from codesign_allocate compiled from source
23:06 < achow101> codesign uses codesign_allocate under the hood somewhere and that's causing us problems
23:06 < jonasschnelli> achow101: thats very interesting... 
23:07 < sipa> jonasschnelli: can you try whag happens if you pass --pagesize 0 to the codesign tool?
23:08 < sipa> that's unrelated, but it may result in smaller ans more predictable signature sizes
23:08 < jonasschnelli> sipa: okay... I executed "./detached-sig-create.sh -s "Bitcoin" --pagesize 0"
23:09 < sipa> no no, to the codesign tool
23:09 < sipa> so inside detached-sig-create.sh
23:09 < achow101> sipa: I've figured out what that codesign source I found doesn't have a call to codesign_allocate - the actual signing is way deeper in the stack and isn't open source. the code I found is just the wrapper to expose things as a command line tool
23:09 < jonasschnelli> sipa: okay
23:09 < sipa> or does it pass everything through?
23:09 < achow101> passing extra args to detached-sig-create should pass through to codesign
23:09 < sipa> ok
23:10 < sipa> achow101: yeah, i tried browsing but i couldn't even find any place where the pagesize argument was used
23:11 < jonasschnelli> https://bitcoin.jonasschnelli.ch/signature-osx_pagesize_0.tar.gz
23:11 < jonasschnelli> I changed L26 to ${CODESIGN} --pagesize 0 -f --file-list ${TEMPLIST} "$@" "${BUNDLE}"
23:12 < jonasschnelli> but why was rc2 fine and rc3 isn't?
23:12 < achow101> beats me
23:12 < achow101> I think we've found an edge case
23:12 < jonasschnelli> as I suspected
23:13 < achow101> it's weird that this is not present in the open soure build though
23:13 < sipa> jonasschnelli: it appears that the macos tool uses a different segment alignment
23:13 < sipa> as your signed result has an alignment of 8192 or 16384 bytes
23:13 < sipa> while the gitian result is 4096 byte aligned
23:14 < jonasschnelli> okay
23:14 < sipa> but the apple (open source) definitely uses 4096
23:14 < sipa> it may be that rc2 just accidentally was already 8192 aligned
23:14 < sipa> we could check
23:14 < sipa> jonasschnelli: how big is the .sign file now?
23:15 < achow101> sipa: I checked earlier, it's 7b0000
23:15 < sipa> ok, i have no explanation then :)
23:16 < jonasschnelli> sipa: jonasschnelli  staff  18496 12 Dez 08:10 Bitcoin-Qt.sign
23:17 < sipa> much better!
23:18 < sipa> the previous ones were 200 kB or so
23:18 < jonasschnelli> I can try to gbuild the macOS signer with that
23:18 < sipa> it's worth a try
23:19 < sipa> if it was a weird edge case due to alignments, maybe this is enough to fix it
23:26 -!- kristapsk_ [~KK@gateway/tor-sasl/kristapsk] has quit [Remote host closed the connection]
23:26 -!- kristapsk_ [~KK@gateway/tor-sasl/kristapsk] has joined #bitcoin-core-dev
23:38 < achow101> --pagesize 0 might actually make it possible for us to hack together a tool for codesigning
23:39 < achow101> that doesn't require macOS
23:39 < jonasschnelli> pagesize 0 seems also not to work
23:39 < jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed_pagesize0.dmg
23:41 < achow101> I was going to suggest that we try using the macports codesign_allocate, but the manpage for codesign says "The system will not accept such substitutes unless they are specially signed (by Apple)." on that topic, so I don't think it will work
23:42 < sipa> indeed
23:42 < sipa> jonasschnelli: same problem?
23:42 < sipa> achow101: is it again the vmsize being off?
23:42 < achow101> checking
23:43 < sipa> jonasschnelli: is that file the signed binary that comes out of the codesign tool?
23:44 < jonasschnelli> sipa: no... that file above is the gitian signer result with using the signature.tar.gz that was built with pagesize 0
23:44 < achow101> i need to compare against the result of the codesign tool
23:44 < sipa> jonasschnelli: we'll also need the codesign output
23:44 < jonasschnelli> okay.. one sec
23:46 < jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-0.21.0rc3-osx-unsigned-pagesize-0.zip
23:46 -!- alko89 [~alko89@unaffiliated/alko89] has quit [Quit: ZNC 1.7.5 - https://znc.in]
23:46 -!- alko89 [~alko89@unaffiliated/alko89] has joined #bitcoin-core-dev
23:47 < achow101> yes same problem
23:48 < achow101> codesign_allocate really likes 7c000 for some reason
23:48 < sipa> achow101: what are the vmsize values now?
23:48 < sipa> wait what
23:48 < sipa> what is the filesize field and the vmsize field in both?
23:48 < achow101> gitian is 0x4800 and codesign is 7c000
23:49 < sipa> dafuq
23:49 < achow101> fileoff and filesize are the same
23:49 < achow101> both are otherwise identical except for vmsize
23:49 < sipa> and what is that filesize?
23:49  * jonasschnelli throws away his mac
23:49 < achow101> filesize 293328
23:49 < sipa> jonasschnelli: a just decision
23:50 < achow101> signature is 18496
23:50 < sipa> 0x48000 is the correct vmsize
23:51 -!- alko89 [~alko89@unaffiliated/alko89] has quit [Client Quit]
23:51 < sipa> well, "correct", from my understanding of what the opensource allocate tool does
23:51 < achow101> yes
23:51 < sipa> and that's no surprise, because that's also what it did
23:51 -!- alko89 [~alko89@unaffiliated/alko89] has joined #bitcoin-core-dev
23:51 < sipa> but so this means it is not a rounding issue at all
23:52 < sipa> unless it's rounding up to a multiple of 0x7c000 !
23:53 < sipa> if you on macos run codesign_allocate infile -a x86_64 18496 -o outfile on the unsigned binary, does it also produce 0x7c000 as vmsize?
23:54 < jonasschnelli> let me try
23:55 < sipa> jonasschnelli: did you accidentally codesign an already-signed binary?
23:55 < jonasschnelli> sipa:.... hmm.. that could be for the latest just done build
23:55 < sipa> vmsize is not expected to go down in that case
23:56 < sipa> so you have to start with the unsigned binary
23:56 -!- sr_gi [~sr_gi@80.174.218.168.dyn.user.ono.com] has quit [Read error: Connection reset by peer]
23:56 < achow101> vmsize 0x0000000000043188
23:56 -!- sr_gi [~sr_gi@80.174.218.168.dyn.user.ono.com] has joined #bitcoin-core-dev
23:56 < jonasschnelli> I try it again with pagesize 0 and make sure it's not signed already
23:56 < achow101> that's not a multple of 0x1000, 0x2000, or 0x4000
23:56 < sipa> achow101: wut
23:57 < achow101> ah wrong file
23:57 < achow101> vmsize 0x0000000000048000
23:57 < achow101> that's correct and what we expect
23:58 < sipa> achow101: with the macos tool?
23:58 < achow101> yes
23:59 < sipa> and just to be sure... if you pass 225312 as size?
--- Log closed Sat Dec 12 00:00:08 2020