--- Log opened Fri Aug 24 00:00:28 2018 --- Day changed Fri Aug 24 2018 00:00 -!- ken2812221 [~ken281222@1-160-140-87.dynamic-ip.hinet.net] has quit [Remote host closed the connection] 00:00 -!- ken2812221 [~ken281222@1-160-140-87.dynamic-ip.hinet.net] has joined #bitcoin-core-dev 00:01 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Ping timeout: 252 seconds] 00:03 -!- Krellan [~Krellan@2601:640:4000:9258:e42b:3557:fd76:95fa] has joined #bitcoin-core-dev 00:05 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has quit [Ping timeout: 255 seconds] 00:08 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 00:09 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 00:15 -!- csknk [~csknk@unaffiliated/csknk] has joined #bitcoin-core-dev 00:19 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has joined #bitcoin-core-dev 00:22 -!- Guyver2 [AdiIRC@guyver2.xs4all.nl] has joined #bitcoin-core-dev 00:33 -!- ken2812221 [~ken281222@1-160-140-87.dynamic-ip.hinet.net] has quit [Ping timeout: 252 seconds] 00:41 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Remote host closed the connection] 00:41 -!- ken2812221 [~ken281222@1-160-140-87.dynamic-ip.hinet.net] has joined #bitcoin-core-dev 00:44 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #bitcoin-core-dev 01:02 -!- IGHOR [~quassel@93.178.216.72] has joined #bitcoin-core-dev 01:06 -!- setpill [~setpill@unaffiliated/setpill] has joined #bitcoin-core-dev 01:06 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 01:08 -!- Ylbam_ [uid99779@gateway/web/irccloud.com/x-yjvlnddzvqmzhkep] has joined #bitcoin-core-dev 01:09 -!- Ylbam_ [uid99779@gateway/web/irccloud.com/x-yjvlnddzvqmzhkep] has quit [Client Quit] 01:11 -!- Krellan [~Krellan@2601:640:4000:9258:e42b:3557:fd76:95fa] has quit [Remote host closed the connection] 01:12 -!- Krellan [~Krellan@2601:640:4000:9258:e42b:3557:fd76:95fa] has joined #bitcoin-core-dev 01:14 -!- ylbam [uid99779@gateway/web/irccloud.com/x-ozxfeoqxoblxlxbe] has joined #bitcoin-core-dev 01:26 * luke-jr wonders why gitian builds seem to ignore cached depends sometimes 01:32 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 01:40 -!- moxuan [~moxuan@39.155.221.55] has quit [Quit: Leaving] 01:40 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 01:40 -!- Zenton [~user@195.235.96.150] has joined #bitcoin-core-dev 01:40 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has quit [Ping timeout: 264 seconds] 01:45 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 02:04 -!- promag [~promag@83.223.249.27] has joined #bitcoin-core-dev 02:08 -!- zivl [~zivl@2601:19a:837f:e4e1:c9ba:4a07:cd88:b9ae] has joined #bitcoin-core-dev 02:09 -!- vexbuy [~vexbuy@46.166.142.215] has joined #bitcoin-core-dev 02:31 -!- vexbuy [~vexbuy@46.166.142.215] has quit [Remote host closed the connection] 02:32 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 02:51 -!- vexbuy [~vexbuy@46.166.142.215] has joined #bitcoin-core-dev 02:51 -!- vexbuy [~vexbuy@46.166.142.215] has quit [Remote host closed the connection] 02:52 -!- vexbuy [~vexbuy@46.166.142.215] has joined #bitcoin-core-dev 02:59 -!- rex4539 [~rex4539@2a02:587:3516:600:dc14:db78:2211:85e7] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 03:20 -!- Krellan [~Krellan@2601:640:4000:9258:e42b:3557:fd76:95fa] has quit [Read error: Connection reset by peer] 03:22 -!- Krellan [~Krellan@2601:640:4000:9258:e42b:3557:fd76:95fa] has joined #bitcoin-core-dev 03:32 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has joined #bitcoin-core-dev 03:55 -!- rex4539 [~rex4539@62.38.35.218] has joined #bitcoin-core-dev 04:07 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 04:08 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 04:12 -!- ken2812221 [~ken281222@1-160-140-87.dynamic-ip.hinet.net] has quit [Ping timeout: 244 seconds] 04:37 -!- spinza [~spin@155.93.246.187] has joined #bitcoin-core-dev 05:01 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has quit [Ping timeout: 276 seconds] 05:06 -!- promag [~promag@83.223.249.27] has quit [Remote host closed the connection] 05:13 -!- berndj [~berndj@azna.co.za] has quit [Quit: ZNC - http://znc.in] 05:16 -!- berndj [~berndj@azna.co.za] has joined #bitcoin-core-dev 05:24 -!- ken2812221 [~ken281222@110.50.144.85] has joined #bitcoin-core-dev 05:39 -!- vexbuy [~vexbuy@46.166.142.215] has quit [Remote host closed the connection] 05:49 -!- Guyver2 [AdiIRC@guyver2.xs4all.nl] has quit [Quit: Going offline, see ya! (www.adiirc.com)] 05:51 -!- vexbuy [~vexbuy@46.166.142.215] has joined #bitcoin-core-dev 05:54 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has joined #bitcoin-core-dev 05:57 -!- ylbam [uid99779@gateway/web/irccloud.com/x-ozxfeoqxoblxlxbe] has quit [Quit: Connection closed for inactivity] 05:58 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 06:02 -!- setpill [~setpill@unaffiliated/setpill] has quit [Quit: o/] 06:37 < cfields> luke-jr: rc2 required rebuilding qt to fix determinism. Is that what you're seeing? 06:38 < cfields> jonasschnelli: macOS detached signature reminder :) 07:15 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds] 07:17 < MarcoFalke> ken2812221: A new section in developer notes? 07:18 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 07:21 < luke-jr> cfields: I'm seeing all the deps being rebuilt for basically the same commit 07:21 -!- vexbuy [~vexbuy@46.166.142.215] has quit [Ping timeout: 272 seconds] 07:29 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 07:32 -!- jhfrontz [~Adium@cpe-184-57-118-36.columbus.res.rr.com] has joined #bitcoin-core-dev 07:37 -!- vexbuy_ [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 07:37 -!- Krellan [~Krellan@2601:640:4000:9258:e42b:3557:fd76:95fa] has quit [Read error: Connection reset by peer] 07:38 -!- vexbuy [~vexbuy@109.201.133.24] has joined #bitcoin-core-dev 07:38 -!- Krellan [~Krellan@2601:640:4000:9258:e42b:3557:fd76:95fa] has joined #bitcoin-core-dev 07:39 -!- vexbuy [~vexbuy@109.201.133.24] has quit [Client Quit] 07:39 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #bitcoin-core-dev 07:39 < jonasschnelli> cfields: oh. One sec 07:41 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has quit [Ping timeout: 264 seconds] 07:44 < cfields> luke-jr: hmm, yea, that's not right 07:45 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Read error: Connection reset by peer] 07:45 < jonasschnelli> cfields: done -> https://github.com/bitcoin-core/bitcoin-detached-sigs/pull/11 07:45 < luke-jr> cfields: is there any way to get debug output to see why it's not using the cached builds? 07:45 < cfields> jonasschnelli: thanks! 07:46 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev 07:46 < cfields> luke-jr: sure, but gitian makes it kinda tough. Can you drop to a shell in the builder? 07:47 < cfields> luke-jr: first thing, try building one more time and seeing if it happens again 07:47 < cfields> IIRC bionic's glibc was updated this/last week, so that might've triggered it 07:49 < luke-jr> oh 07:52 < luke-jr> pretty sure I've seen it multiple times in the last few days, but I haven't been watching closely enough to be sure 07:52 < cfields> (depends does something like COMPILER_SEED=`$CC --version --verbose` 07:52 < luke-jr> so I'll do a few more just in case 07:52 < cfields> ) 07:52 < cfields> and if it changes, it invalidates all current builds 07:52 < cfields> ok 07:53 < luke-jr> (also, the cache files are replaced with the same filename) 07:53 < cfields> oh, that's weird 07:53 < cfields> the filename contains the hash of the determined build recipe. So that shouldn't be happening. 07:54 < cfields> maybe some filesystem thing keeps them from sending correctly to the builder? 07:55 < luke-jr> no sign of trouble there that I could tell 07:56 < luke-jr> hmm 07:57 < luke-jr> I have cache/bitcoin-linux-0.17/bitcoin-linux-0.17/, possibly from manual copy-from-target on failures 07:57 < luke-jr> it's possible since the glibc bump, I've only had failures.. 07:57 < cfields> hmm? 07:58 < luke-jr> cfields: when gitian fails, it doesn't copy cache changes back to the host; I'd intended to manually do that, but it's possible they ended up in the wrong place 07:58 < luke-jr> combined with the glibc update you mentioned, that may explain why it's rebuilding every time 07:58 < cfields> ah, makes sense 08:03 -!- keymone [~keymone@ip1f12c093.dynamic.kabel-deutschland.de] has joined #bitcoin-core-dev 08:04 -!- jamesob [~james@100.38.11.146] has joined #bitcoin-core-dev 08:11 < cfields> gitian builders: detached sigs for 0.17.0rc2 are up 08:11 -!- as1nc [~as1nc@ax213-1-82-66-157-194.fbx.proxad.net] has joined #bitcoin-core-dev 08:11 -!- MaxHastings_ [47ef9337@gateway/web/freenode/ip.71.239.147.55] has joined #bitcoin-core-dev 08:11 -!- Dizzle [~dizzle@108.171.182.16] has joined #bitcoin-core-dev 08:12 -!- rex4539 [~rex4539@62.38.35.218] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…] 08:13 < cfields> fanquake/achow101/jonasschnelli/provoostenator/luke-jr/wumpus/MarcoFalke/fivepiece: ^^ 08:13 < jonasschnelli> /o/ 08:14 < MaxHastings_> Hey. I just have a quick question. Are all the known vulnerabilities for SPV wallets listed here? https://bitcoin.org/en/developer-guide#simplified-payment-verification-spv I've been told on the Bitcoin slack that there are more. 08:18 < as1nc> Hello guys, just published a 2018 simple guide to set up a bitcoin core node on, feedback really appreciated. https://gist.github.com/larafale/b4df34a97c7134cf1579539caf2db2c2 08:19 < jonasschnelli> MaxHastings_: does it mention that the current used bloomfilter false-positive-rates do not protect privacy? (See Jonas Nicks master thesis from 2015) 08:19 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has joined #bitcoin-core-dev 08:20 < jonasschnelli> MaxHastings_: there are eventually non-protocol vulnerabilities (implementation issues) 08:20 < jonasschnelli> as1nc: nice. You could mention pruning? 08:21 < jonasschnelli> (and or electrum personal server) 08:21 < jonasschnelli> But its OT in this channel... so use #bitcoin 08:24 < luke-jr> cfields: can you sign PPC Windows builds too? 08:24 < luke-jr> jk ;) 08:25 < cfields> luke-jr: eh? does that exist? 08:25 < luke-jr> cfields: dunno, hence jk 08:25 < cfields> oh, phew 08:25 < luke-jr> XD 08:26 < luke-jr> (working on POWER8+ gitian binaries) 08:26 < luke-jr> (for Linux) 08:26 < cfields> because if they couldn't pull off small-endian arm, BE ppc wouldn't stand a chance :) 08:26 < luke-jr> who said BE? 08:26 < luke-jr> afaik even when Windows did support PPC, it was LE 08:26 < cfields> yeah yeah, that's why I specified 08:26 < cfields> wait, they actually did support PPC at one point? 08:26 < MaxHastings_> jonasschnelli: Ah I see I did not know bloom filters were ineffective for keeping user privacy. Are there any other missing vulnerabilities on that page other than privacy concerns? I was told on Bitcoin slack that the SPV client could be tricked to change its consensus rules by malicious nodes. 08:27 < luke-jr> cfields: NT supported a lot of archs 08:27 < sipa> MaxHastings_: by definition, yes 08:28 < cfields> huh. I used win embedded a few places, so I guess that makes sense. I never would've thought ppc though. 08:28 < sipa> MaxHastings_: they rely on the majority of the hashrate 08:28 < luke-jr> "Windows NT 3.51 added support for the PowerPC processor in 1995, specifically PReP-compliant systems such as the IBM Power Series desktops/laptops and Motorola PowerStack series; but despite meetings between Michael Spindler and Bill Gates, not on the Power Macintosh as the PReP compliant Power Macintosh project failed to ship." 08:29 < luke-jr> not really sure what that means XD 08:29 < as1nc> jonasschnelli, yes but i really want to incite people to txindex their chain so they can benefit the full spectrum of bitcoin capabilities. lightning for exemple require a txindexed chain right ? 08:30 < sipa> as1nc: i don't think so (or that's temporary in any case) 08:30 < jonasschnelli> Not sure. C-Lightning doesn't IMO 08:30 < as1nc> lnd does i think 08:31 < sipa> i'm sure that's temporary until better protocols to communicate exist 08:32 < sipa> (like bip157) 08:34 < MaxHastings_> sipa: Good to know. 08:37 < gmaxwell> c-lightning doesn't. 08:37 < gmaxwell> as1nc: txindex is non-viable long term. 08:37 < gmaxwell> Anytime you think "X _requires_ txindex" you should think "X will eventually force people to depend on a centeralized service". 08:38 < molz> lnd doesn't require txindex anymore 08:40 -!- Zenton [~user@195.235.96.150] has quit [Ping timeout: 264 seconds] 08:41 < as1nc> ok interesting, I'm txindexing because I want to provide services for people. but the only case i find indexing worth it for personnal use is SPV wallets. does it makes sense ? 08:41 < as1nc> i use electrum pointed to my node 08:41 < sipa> as1nc: txindex is purely a local feature; it is not exposed over the network 08:43 < as1nc> sipa, yes 08:44 -!- phwalkr [~phwalkr@2001:1284:f019:69b1:5cc7:6413:ac59:30c5] has joined #bitcoin-core-dev 08:47 -!- phwalkr [~phwalkr@2001:1284:f019:69b1:5cc7:6413:ac59:30c5] has quit [Remote host closed the connection] 09:06 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 09:07 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 09:09 < Dizzle> as1nc: yep, it makes sense, for both personal use or providing centralized services. As for lnd, if you don't have a txindex, lnd basically falls back to rescanning blocks when it needs to know a tx's status, so a bit of a performance drag in some cases. 09:10 -!- rex4539 [~rex4539@ppp-2-87-178-187.home.otenet.gr] has joined #bitcoin-core-dev 09:11 < gmaxwell> how is scanning blocks going to be a performance drag? thats how a node works anyways. 09:21 < Dizzle> gmaxwell: at lnd level, it's done manually instead of regularly i.e. specifically to find what may have befallen the tx - https://github.com/lightningnetwork/lnd/blob/c9bead7c21b1b83d69894935f4e1269ceffbecc7/chainntnfs/bitcoindnotify/bitcoind.go#L535 - so instead of a quick index lookup in Core or btcd, it's back and forth RPC and comparison looping. 09:25 < as1nc> Dizzle: thx, so it make sense to txindex your chain for personnal use, even if, like gmaxwell said, its not viable long run 09:32 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 09:34 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 09:39 -!- phwalkr [~phwalkr@2001:1284:f01c:5d1e:64d0:7be4:35c3:7e85] has joined #bitcoin-core-dev 09:42 < gmaxwell> Dizzle: I don't see any back and forth there, it just gets the block. 09:43 < gmaxwell> txindex lowers performance a lot, so I'm still not seeing how you're saying the lnd performance is worth without it. 09:44 < luke-jr> if only bech32 addresses were shorter, we could have called it belch32 09:44 -!- phwalkr [~phwalkr@2001:1284:f01c:5d1e:64d0:7be4:35c3:7e85] has quit [Ping timeout: 264 seconds] 09:52 -!- peevsie [~peevsie@2604:2000:f18f:e300::2] has joined #bitcoin-core-dev 09:55 < Dizzle> gmaxwell: only saying the performance is worse without txindex for the cases when historical lookup of a tx's status is needed. I mean, that's what the index is useful for. The back and forth I'm referring to is getblockhash+getblock RPC calls for each height in turn, check for a match, repeat. 10:01 < gmaxwell> Dizzle: and my point is that you're missing the forrest for the trees. fetching the list of transactions in a block is cheap. maintaining the txindex slows down block processing a lot. 10:02 < gmaxwell> So I think what you're describing costs several milliseconds per block to save a millisecond on some blocks. 10:06 -!- Urgo [~Urgo@cpe-107-15-142-254.nc.res.rr.com] has joined #bitcoin-core-dev 10:10 < luke-jr> how does symbol-check not complain about aarch using symbols from glibc 2.17? 10:10 -!- peevsie [~peevsie@2604:2000:f18f:e300::2] has quit [Ping timeout: 252 seconds] 10:14 < luke-jr> …oh, gitian disables it for aarch XD 10:18 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds] 10:26 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection] 10:30 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Read error: Connection reset by peer] 10:31 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev 10:36 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 10:41 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 272 seconds] 10:42 -!- Zenton [~user@54.104.135.37.dynamic.jazztel.es] has joined #bitcoin-core-dev 10:48 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 10:50 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 10:53 -!- esotericnonsense [~esotericn@unaffiliated/esotericnonsense] has quit [Ping timeout: 272 seconds] 10:53 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Read error: Connection reset by peer] 10:59 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 11:02 < jnewbery> gmaxell: "txindex lowers performance a lot" - is that still true after #13033 ? 11:02 < gribble> https://github.com/bitcoin/bitcoin/issues/13033 | Build txindex in parallel with validation by jimpo · Pull Request #13033 · bitcoin/bitcoin · GitHub 11:03 -!- esotericnonsense [~esotericn@unaffiliated/esotericnonsense] has joined #bitcoin-core-dev 11:08 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Ping timeout: 268 seconds] 11:08 < gmaxwell> jnewbery: maybe it is no longer "a lot", but it will still be more costly than checking the txids in a block, since it does that and a lot more. 11:09 < gmaxwell> jnewbery: I missed that PR, so thanks for pointing that otu. 11:09 < BlueMatt> I mean I dunno what the IOPS cost of it is, but if you're running with huge dbcache it should be almost free modulo the fact that we fsync the block files all the time (which we should stop doing, really) 11:10 < sipa> BlueMatt: once per hour 11:10 < sipa> and only the ones written to afaik 11:10 -!- BashCo_ [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 11:10 < BlueMatt> not for block files, no, every time we move to a new one 11:10 < BlueMatt> last I checked 11:11 < BlueMatt> yea, every time we move to a new file we FlushBlockFile 11:11 < BlueMatt> which fsync's, afaics 11:11 < gmaxwell> BlueMatt: your comment is frustrating me. Above we are having a discussion where the argument was made that a whole txindex should be enabled to replace simply checking each block for a transaction. 11:11 < BlueMatt> ah, I missed context, sorry 11:11 < BlueMatt> yea, obviously not 11:11 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [Ping timeout: 252 seconds] 11:12 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #bitcoin-core-dev 11:12 < gmaxwell> perhaps txindex no longer slows down reindex by 20% or whatever. but yea... 11:12 < sipa> performance isn't really the reason not to use it - it just doesn't scale well 11:12 < BlueMatt> anyway, as a separate point, spinning-rust-ibd-with-huge-dbcache the hang after each "Leaving block file" entry is really annoying 11:12 < BlueMatt> sipa: well, *also* performance :p 11:18 -!- BashCo_ [~BashCo@unaffiliated/bashco] has quit [Ping timeout: 252 seconds] 11:19 -!- Krellan [~Krellan@2601:640:4000:9258:e42b:3557:fd76:95fa] has quit [Remote host closed the connection] 11:22 -!- BashCo [~BashCo@unaffiliated/bashco] has joined #bitcoin-core-dev 11:23 -!- as1nc [~as1nc@ax213-1-82-66-157-194.fbx.proxad.net] has quit [Quit: Leaving] 11:23 < BlueMatt> sipa: wait, isnt extended-encoding the *only* way you can encode a 0-input 1-output transaction? 11:23 < BlueMatt> reliably, at least 11:23 < BlueMatt> short of psbt 11:25 < sipa> BlueMatt: there is no reliable way to encode 0-input transactions 11:25 < BlueMatt> sure, if you use extended-encoding 11:26 < sipa> that's still potentially ambiguous 11:26 < BlueMatt> the use-all-data heuristic should handle it fine 11:26 < BlueMatt> and doesnt-need-to-be-backwards-compatible APIs (eg rust-bitcoin's deserializer) can *only* handle it that way 11:27 < sipa> i'm confused 11:27 < sipa> it is not legal to encode a tx without witnesses using extended encoding 11:27 < sipa> so there should never be any ambiguity 11:27 < BlueMatt> but every deserializer I've ever seen handles it fine, and its the only way to have no ambiguity in decoding 0-input txn 11:27 < BlueMatt> so I think we should revise the spec to match implementations? 11:28 < BlueMatt> or am I confusing myself 11:28 -!- leishman [~leishman@50.237.29.22] has joined #bitcoin-core-dev 11:28 < sipa> sigh, so make an exception to the only-extended-if-has-witness rule in case there are 0 inputs? 11:29 < sipa> that's fine by me - i mostly care about only having a single encoding for actual transactions 11:29 < sipa> using the raw tx format for partial transactions was always a hack at best 11:29 < BlueMatt> yes, agreed, but the only reliably hack is always-use-extended, sadly :( 11:30 < BlueMatt> but, ok, fair, only doing it for 0-inputs seems reasonable to me 11:30 < sipa> we can also make the serializer do the same 11:30 < sipa> always use extended encoding when 0 inputs 11:30 < BlueMatt> would seem prudent to me :( 11:30 < sipa> ... or psbt 11:30 -!- BashCo [~BashCo@unaffiliated/bashco] has quit [] 11:30 < sipa> but everything at its time 11:30 < BlueMatt> yea, well hopefully people migrate to psbt 11:30 < BlueMatt> yea 11:30 -!- andytoshi [~apoelstra@unaffiliated/andytoshi] has joined #bitcoin-core-dev 11:37 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 11:40 -!- leishman [~leishman@50.237.29.22] has quit [Remote host closed the connection] 11:41 -!- Empact [~empact@192-195-80-226.PUBLIC.monkeybrains.net] has joined #bitcoin-core-dev 11:42 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Ping timeout: 252 seconds] 11:45 < andytoshi> achow101: how do you use PSBT when nobody knows the full set of inputs up front? 11:46 < andytoshi> in particular, can you use PSBT to avoid the 0-input serialization ambiguity 11:46 < sipa> right 11:46 -!- SopaXorzTaker [~SopaXorzT@unaffiliated/sopaxorztaker] has quit [Remote host closed the connection] 11:47 < sipa> for 0-input things people should just use a list of amounts/scriptpubkeys 11:47 < sipa> (i've mentioned extended psbt to support 0 inputs just as a container format, though) 11:48 < andytoshi> if we're going to propose an extension BIP to add p2c fields and industry/private reserved IDs, we should throw that in the pile 11:49 < sipa> in any case, PSBT's serialization supports 0 inputs fine 11:49 < sipa> (because it specifies using non-witness serialization for the tx, there is never any ambiguity) 11:49 < andytoshi> oh right 11:50 < andytoshi> ok i forgot that, never mind me, sorry 11:50 < sipa> it's not really usable to use for the various PSBT roles just because there isn't anything to be done with it 11:50 < gmaxwell> https://bitcoinperf.com/timeline/#/?exe=3,4,2,1&base=1+23&ben=reindex.522000.dbcache=2048.mem-usage&env=1&revs=50&equid=off&quarts=on&extr=on < I think this should be blocking 0.17. 11:50 < gmaxwell> 20% memory usage increase for unknown reasons isn't cool. 11:51 < BlueMatt> sipa: wait, so psbt is not usable with the existing deserializer? 11:52 < sipa> BlueMatt: yes it is 11:52 < BlueMatt> cause 0-input non-witness is ambiguous? 11:52 < jamesob> gmaxwell: still bisecting 11:52 < sipa> BlueMatt: not if it is known to be non-witness 11:52 < BlueMatt> unless you give it a flag saying "it is 0-input, and not-witness-encoded, i promise" 11:52 < BlueMatt> ah, i guess we have that flag, other deserializers need it, though 11:52 < sipa> BlueMatt: which is what the implementation in core does 11:52 < BlueMatt> which is super obnoxious 11:52 < jamesob> running some tests to ensure increased mem usage isn't some quirk with the benchmarking framework 11:53 < sipa> BlueMatt: well, deal with it 11:54 < BlueMatt> lolk, but why doesnt it just serialize a list of outputs? 11:54 < sipa> BlueMatt: because it needs to serialize inputs too 11:55 < BlueMatt> I mean in no-input cases 11:55 < sipa> PSBT isn't designed for no-input cases 11:55 < BlueMatt> I thought psbt was supposed to simplify the deserialization ambiguity, not work around it with more flags :( 11:55 < sipa> it's to simplify the signing procedure 11:55 < sipa> BlueMatt: i don't understand why this is hard 11:55 < BlueMatt> its not "hard", just really annoying 11:55 < sipa> you know there is no witness, don't try to deserialize as a witness 11:56 < BlueMatt> yes, well I presume most deserializers dont have a flag for that 11:56 < BlueMatt> cause they dont need it 11:56 < BlueMatt> at least rust-bitcoin currently doesnt 11:56 < sipa> ? 11:56 < BlueMatt> which means plumbing through a ton of flags stuff 11:56 < sipa> how do you deal with a tx from a non-witness peer? 11:56 < gmaxwell> jamesob: well I also recently noticed the infinite dbcache case using much more dbcache size than I remember. 11:56 < BlueMatt> you never see 0-input txn on peer network? 11:56 < BlueMatt> so no ambiguity there 11:57 < sipa> yes, but you should reject if it were to contain a witness 11:57 < jamesob> gmaxwell: ah okay, so that corroborates 11:57 < sipa> i guess that can be done after deserialization, fine 11:57 < sipa> but really, is this worth arguing over? 11:57 < sipa> deserialization of 0-input is a pain, i'm sorry for that 11:58 < BlueMatt> I mean is it too late to fix the psbt spec? its just annoying to have so many serialization/deserialization modes 11:58 < sipa> yes, it is far too late 11:58 < BlueMatt> if the thing already has a "this is a 0-input tx" indicator, then why not have just used a list of outputs instead 11:58 < BlueMatt> plus nVserion/nLockTime 11:58 < sipa> no it doesn't have a this is a 0-input tx 11:59 < sipa> that indicator is the lack of inputs 11:59 < BlueMatt> ehh, whatever, I guess can be dealt with 11:59 < sipa> and again, psbt isn't designed for 0 inputs; it's designed to simplify coordination of signing, which by definition needs at least 1 input 11:59 < sipa> it just so happens that it is unambiguous even with 0 inputs 12:00 < gmaxwell> sipa: uh, isn't "I create a PSBT with the outputs I want, and give it to you so you can add inputs (and your change)" a supported use case 12:00 < achow101> BlueMatt: there are no flags in psbt. the global unsigned tx is always non-witness 12:00 < achow101> the input txs are self descriptive as to witness or not as they are always valid network txs 12:01 < sipa> gmaxwell: not initially no, but there is no reason why it couldn't 12:01 < BlueMatt> achow101: wait, huh? 0-input isnt "valid network txs" 12:01 < achow101> BlueMatt: input txs are txs that already exist on the network .they cannot be 0-input 12:01 < gmaxwell> BlueMatt: he's talking about prevout inputs. 12:01 < sipa> BlueMatt: the inputs 12:01 < gmaxwell> for checking fees. 12:01 < BlueMatt> ah 12:02 < BlueMatt> anyway, I'll stop complaining since I obviously didnt read the spec in detail 12:02 < sipa> gmaxwell: in PSBT role terminology that would be someone who takes the output of a Creator, and then "creates" a new PSBT for a different transaction 12:02 < sipa> gmaxwell: but in the workflow, all steps operate on the same "proposed transaction" 12:03 < sipa> when there are 0 inputs, there isn't really a proposed transaction 12:03 < sipa> though the same format could easily be used to support that 12:04 < achow101> gmaxwell: psbt's workflows are currently only defined for the signing stage. it assumes that all inputs and outputs have been decided and negotiated in some previous step 12:04 < BlueMatt> ah, so there are never any 0-input txn encoded in it at all? 12:04 < sipa> BlueMatt: in the PSBT workflow, no 12:04 < achow101> but you could still use psbt for negotiating the inputs. I'm not sure why you would though 12:05 < BlueMatt> sipa: why didnt you say that in the first place? :) 12:05 < sipa> but the format can be used for that, if you want 12:05 < achow101> BlueMatt: no, there is not. but it technically still works and will be deserialized properly 12:05 < sipa> 18:55:17 < sipa> PSBT isn't designed for no-input cases 12:05 < sipa> 18:55:35 < sipa> it's to simplify the signing procedure 12:07 < sipa> it's kind of annyong to think about PSBT-the-container-format (which supports 0 input fine, if you're willing to do the decode-knowing-there-is-no-witness thing) and PSBT-the-procedure-for-signing (which clearly doesn't) as separate 12:08 < BlueMatt> well I mean tell me to shut up cause I havent read it in detail, but I think in a new container-format we should be encoding 0-input with extended serialization 12:09 < BlueMatt> cause thats what we just said, above, too 12:09 < sipa> i think that's insane. 12:10 < sipa> PSBT implementations aren't even required to support segwit 12:10 < achow101> BlueMatt: why? there are no witnesses to serialize ever 12:10 < sipa> just add a flag to your deserializer 12:10 < achow101> BlueMatt: a psbt has two types of txs: the unsigned tx (the one you want to sign) and the input txs (ones already on the network). 12:10 < achow101> the unsigned tx, by definition, has no signatures. therefore it never has witnesses 12:11 < achow101> thus it is completely unnecessary to serialize the unsigned tx with extended serialization as it can never have a witness 12:11 < BlueMatt> sipa: we just discussed, above, encoding 0-input with extended serialization, since thats the *only* way to do it that is non-ambiguous 12:11 < BlueMatt> and otherwise createrawtransaction+fundrawtransaction are...not ideal 12:11 < gmaxwell> I'm just really confused by this discussion. I expected that PSBT was the fix for the createrawtransaction ambiguity case. 12:12 < gmaxwell> and now you're saying that it isn't but it is? 12:12 < achow101> BlueMatt: it is completely unambiguous because the unsigned tx is always non-witness 12:12 < gmaxwell> so this discussion is pointless. 12:12 < sipa> BlueMatt: this is the *first* time ever in my life i hear the suggestion to use extended serialization for 0 input 12:12 < luke-jr> s/unsigned/no-input/ 12:13 < sipa> BlueMatt: it's a cool trick, i admit 12:13 < BlueMatt> sipa: well its either that or we have to pass a flag into signrawtransaction (and any equivalent APIs in any bitcoin library anywhere) 12:13 < sipa> but i had always lived under the assumption that our deserializer failed on that 12:13 < sipa> it was just accidentally removed in a refactor 12:13 < BlueMatt> of which there are many, any I generally dont trust most bitcoin libraries...for obvious reasons 12:13 < achow101> BlueMatt: why are you signing a 0-input tx? 12:13 < achow101> what is there to even sign? 12:13 < sipa> achow101: not the point 12:13 < BlueMatt> sipa: for p2p, it probably should, because, you know, 0-input is bogus on the p2p/consensus layer 12:14 < BlueMatt> achow101: not signing, funding 12:14 < BlueMatt> achow101: right now the suggested workflow of createrawtransaction, fundrawtransaction, signrawtransaction, sendrawtransaction is hit by the ambiguity bug 12:14 < gmaxwell> achow101: See our rpc, "fundrawtransaction" --- that isn't "Fun! Draw Transaction." :P 12:14 < sipa> BlueMatt: createraw and fundraw are merged into one in PSBT workflow 12:14 < BlueMatt> its (mostly) fine right now, but my understanding was the psbt was addressing exactly that use-case but across multiple signers like multisig and the like 12:15 < luke-jr> achow101: you don't know it's zero input yet 12:15 < gmaxwell> the reason they exist seperately to begin with is that there are workflows where they need to be seperate. E.g. coinjoins. 12:15 < sipa> FFS people 12:15 < BlueMatt> sipa: fair, but you just mentioned that the existing container could be extended to split that out, and my point is, "is the container well-defined for the ambiguity bug" 12:15 < gmaxwell> or partial coincontrols. ("spend _this_ coin plus, whatever is needed") 12:15 < sipa> can you have commented on this 9 months ago? 12:16 < sipa> BlueMatt: YES IT IS. NO WITNESSES IN THE UNSIGNED PSBT TRANSACTION 12:16 < sipa> i'm tired of this discussion 12:16 < gmaxwell> sipa: I read the spec and saw no evidence that 0 inputs wasn't a supported case! 12:16 < sipa> the fact that you want to reuse your heuristic decoder for raw transactions in PSBT may mean you need to add a flag to disable that feature 12:16 < sipa> in a post-PSBT world, nobody ever needs to implement such a heuristic again 12:17 < achow101> "Type: Unsigned Transaction PSBT_GLOBAL_UNSIGNED_TX = 0x00 ... The transaction must be in the old serialization format (without witnesses)" 12:17 < sipa> because actual transactions will always have witnesses (or use unambiguously old serialization for things without) 12:17 < sipa> and PSBTs will never have witnesses 12:17 < gmaxwell> it's surprising to me that you and achow didn't except PSBT to be a superset of the applications of the existing raw transactions. 12:17 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has quit [Ping timeout: 276 seconds] 12:18 < gmaxwell> achow101: great, and the old tx format supports 0 inputs. 12:18 < achow101> gmaxwell: exactly 12:18 < sipa> gmaxwell: not... really 12:18 < achow101> gmaxwell: but it is _always_ the old serialization. no ambiguity there 12:19 -!- MaxHastings_ [47ef9337@gateway/web/freenode/ip.71.239.147.55] has quit [Quit: Page closed] 12:19 < sipa> this discussion only happened because someone only has a heuristic tx decoder in their software, and finds it annoying to add a way to disable that heuristic 12:19 < gmaxwell> achow101: sure, I agree. Sipa was asking why wasn't this commented on 9 months ago. And my reply is because the shortfall (that 0 input PSBT aren't a supported usecase) isn't something I could extract from the spec. 12:20 < luke-jr> achow101: the ambiguity is because you don't know if it's 0-input or 1+ input segwit until you parse it correctly 12:20 < sipa> luke-jr: you're missing context 12:20 < gmaxwell> luke-jr: it's not ambigius in PSBT. 12:20 < luke-jr> ok 12:20 < gmaxwell> In PSBT the internal tx never has witnesses. 12:21 < achow101> gmaxwell: the format supports 0-inputs just fine. I think there's even a test for it. the processes that we also defined in the bip do not. 12:21 < sipa> gmaxwell: i think i always thought of PSBT as a better unambiguous serialization... but the BIP describes both a serialization and the procedures around it 12:22 < achow101> but the processes don't matter that much. you can use the psbt format for whatever steps that occur before signing and then use the processes defined in the bip 12:22 < sipa> and those procedures don't really map well to the part where you're still deciding what exactly to sign 12:22 < gmaxwell> I never read that stuff, it's not normative (doesn't change the meaning of the datastructures), and expirence from other BIPs suggests tha people ignore them. (see, for example BIP32) 12:22 < gmaxwell> (or at least don't read it carefully) 12:22 < sipa> gmaxwell: well, then you were right 12:22 < sipa> that part of PSBT supports 0 inputs fine :) 12:33 -!- csknk [~csknk@unaffiliated/csknk] has quit [Quit: leaving] 12:41 < gmaxwell> sipa: thanks! confusion resolved!! 12:47 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 12:53 -!- plankers [~plank@38.87.81.82] has joined #bitcoin-core-dev 13:01 -!- BlueMatt [~BlueMatt@unaffiliated/bluematt] has left #bitcoin-core-dev ["Ex-Chat"] 13:03 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 13:04 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 13:13 < jnewbery> sipa: performance isn't really the reason not to use it - [txindex] just doesn't scale well" - scale well in what regard? Space requirements should be linear with blockchain size for a full archival node, right? 13:15 < sipa> jnewbery: i don't think it is realistic that everyone needs to store the whole chain 13:16 < sipa> or have fast access to it 13:16 < jnewbery> I agree! But for those that do, txindex is just a constant factor increase in space requirements 13:16 < sipa> sure 13:16 < jnewbery> ie txindex scales as well as archival nodes scale 13:17 < sipa> but building solutions that rely on having a fully-indexed blockchain available isn't scalable 13:17 < sipa> you don't need many archival nodes 13:18 < sipa> furthermore, nothing really _needs_ access to the full chain if designed well, in my opinion 13:18 < sipa> except for debugging purposes 13:18 < jnewbery> a block explorer? 13:18 < sipa> for example 13:19 < sipa> (i count that as debug purposes) 13:19 < sipa> not something to rely upon for production 13:20 < jnewbery> I agree that products shouldn't *rely* on block explorers, but there's demand for them now and I can't imagine there'd ever not be demand for them 13:20 < sipa> sadly enough :( 13:22 < sipa> however, in general i think we should discourage building on things like txindex 13:22 < jamesob> and in any case it seems like there'll always be use-cases which require rescan; ie when you become interested in an output you weren't keeping tabs on before 13:23 < sipa> as i fear we may end up with a less scalable ecosystem if everyone assumes it's easy to just look up whatever transaction 13:23 < sipa> and of course there are use cases - it wouldn't be there otherwise 13:24 < sipa> jamesob: i think that's either sign of a badly designed system, or an emergency (for which i think it's fine to need some index, you could also pay someone to look things up for you) 13:25 < jamesob> yeah, maybe so 13:25 -!- roasbeef [~root@104.131.26.124] has joined #bitcoin-core-dev 13:26 < roasbeef> lnd doesn't require the txindex, it'll detect if the full node has it on or not, and fall back to just fetching blocks it needs manually, eventually if the gcs filters are exposed on RPC it could also fetch those before going for the full block it needs 13:29 < jnewbery> It's fair to say that 'assuming that it's easy to just look up whatever transaction' isn't a scalable architecture to build a product on, but I don't think the txindex in itself is unscalable now that we #13033 13:29 < gribble> https://github.com/bitcoin/bitcoin/issues/13033 | Build txindex in parallel with validation by jimpo · Pull Request #13033 · bitcoin/bitcoin · GitHub 13:31 < roasbeef> jnewbery: things could also be split up into distinct database, or even really just like memory mapped direct hash tables, shoving everything in leveldb makes other routine stuff slower as you end up with thousands of files that all need to be memory mapped and also to watch out for fd limits, then on top of that you end up with a handfull of disk seeks due to the file and sstable structure to read something like the offset of a txid in a block 13:33 < jamesob> roasbeef: jimpo's txindex change already moves indices into separate leveldbs 13:33 < roasbeef> Dizzle: we now continually also update a "height hint" so really a checkpoint of the state of that outpoint/txid of its best known state, so like if we know it isn't conf'd at height 100, but it was broadcast at height 10 we know now to start at height 10 13:33 < sipa> roasbeef: BIP157 will improve things further, i assume? 13:33 < Dizzle> roasbeef: nice 13:35 < jnewbery> ok, yes - that is fair, although in practice I'm not sure how much performance impact there is. If it's truly a performance bottleneck, then I assume #13243 gives us the infrastructure to store txindex somewhere else 13:35 < gribble> https://github.com/bitcoin/bitcoin/issues/13243 | Make reusable base class for auxiliary indices by jimpo · Pull Request #13243 · bitcoin/bitcoin · GitHub 13:35 < roasbeef> sipa: yeh, either if it's going over p2p network, or just even getting the filters over rpc from the full node, as it can fetch the filters in batch, and also cache filters on its end 13:35 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has joined #bitcoin-core-dev 13:37 < jamesob> jnewbery: indexes are already isolated from other dbs in /datadir/indexes/[index name] 13:37 < roasbeef> also eventually the stuff about creating larger like hierarchical multi-block filters can help as well when the node has been offline for a long-ish time 13:45 < gmaxwell> Under a no-address reuse assumption (lol), hierarchical filters are never a win. 13:46 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 13:46 < gmaxwell> 13:16:57 < jnewbery> ie txindex scales as well as archival nodes scale 13:46 < gmaxwell> I don't expect that archive nodes will be a thing in 10 years, outside of academic curiousity and what not. 13:47 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 13:47 < gmaxwell> No more than traces of all announced transactions from all peers in the network are a thing now. 13:48 < gmaxwell> I expect that new nodes will sync up using a months old UTXO state thats vallidated according to the assumevalid model, then catch up. 13:52 < jnewbery> Seems possible, but for the next 10 years I expect at least some people will want to run archival nodes! 13:52 < gmaxwell> oh absolutely, and we should do what we can to make that work well too. 13:52 < sipa> i hope they do. 13:52 < sipa> but i hope no services realpy need them 13:53 < gmaxwell> I've pushed a lot on optimizations related to e.g. reindex/ibd in the last year specifically because once we're syncing from AV-UTXO there will be much less motivation to improve these things. 13:54 < jnewbery> to be clear - I'm not arguing that we should *encourage* the use of txindex or needing a full blockchain. I just wanted understand what was meant by "txindex is unscalable". 13:56 < sipa> right; i think it's more "an ecosystem relying on txindex doesn't scale well" 13:56 < jnewbery> yes, makes sense 13:56 < sipa> txindex as an algorithmic thing itself scales as well as it could i think 13:57 < gmaxwell> Well I'm sure it's lacking a ton of possible optimizations. 13:57 < sipa> sure, but no dramatic complexity changes 13:58 < gmaxwell> no, just constant factors. 13:58 < gmaxwell> it's not the txindex structure itself isn't scable, it's that operations over the chain history aren't scalable. 14:00 < jnewbery> sipa, has your opinion changed at all since this: I both hate and love this patch. I hate making it easy to build infrastructure that relies on a fully-indexed blockchain (which shouldn't be necessary), as it may remove the incentive to build more scalable solutions. On the other hand, in cases where the alternative is relying on a trusted third party, this approach is certainly preferable, and 14:00 < jnewbery> would allow an RPC-based blockexplorer, for example. 14:00 < jnewbery> (#2802) 14:00 < gribble> https://github.com/bitcoin/bitcoin/issues/2802 | Add an address-based transaction index by sipa · Pull Request #2802 · bitcoin/bitcoin · GitHub 14:01 < gmaxwell> RPC based explorer also needs a reversed spentness index. 14:03 < gmaxwell> jnewbery: it's an interesting question of if people would actually use it if it was provided. If they would I think it would be good to have. ... at least for the next 10 years, _some_ users would stop bleeding privacy wise. 14:07 -!- Krellan [~Krellan@50-242-94-241-static.hfc.comcastbusiness.net] has joined #bitcoin-core-dev 14:07 < sipa> jnewbery: i have the same love-hate relation with it :) 14:09 < jnewbery> good to know. Thanks! 14:09 < instagibbs> https://github.com/bitcoin/bitcoin/pull/14055 :( another RC? 14:10 < instagibbs> we(not me clearly, someone else) kind of broke the call. Probably is a way around it(walletupdatepsbt?) 14:10 < instagibbs> walletprocesspsbt 14:11 < gmaxwell> Well I think unless it turns out that the memory usage increase is fictional, we'll have another RC anyways. 14:12 < instagibbs> phew 14:16 -!- farmerwampum [~farmerwam@104.129.28.10] has quit [Ping timeout: 252 seconds] 14:33 -!- andytoshi [~apoelstra@unaffiliated/andytoshi] has quit [Ping timeout: 272 seconds] 14:49 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 14:51 -!- marcinja [~marcin@static-100-38-11-146.nycmny.fios.verizon.net] has quit [Quit: Leaving] 14:56 -!- dendisuhubdy [~dendisuhu@keatext-srx100.isp.ip4b.net] has joined #bitcoin-core-dev 14:59 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Ping timeout: 272 seconds] 15:00 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has joined #bitcoin-core-dev 15:05 < luke-jr> woo finally got powerpc64 builds out of gitian working 15:06 < luke-jr> just in time for bluematt to close his PR :x 15:14 -!- dendisuhubdy [~dendisuhu@keatext-srx100.isp.ip4b.net] has quit [Quit: Leaving...] 15:16 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection] 15:19 -!- jungly [~jungly@host5-191-dynamic.42-79-r.retail.telecomitalia.it] has joined #bitcoin-core-dev 15:19 -!- meshcollider_ [uid246294@gateway/web/irccloud.com/x-gydxecdiidffgcuo] has joined #bitcoin-core-dev 15:34 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 244 seconds] 15:35 -!- murrayn [~dafuq@unaffiliated/murrayn] has joined #bitcoin-core-dev 15:36 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev 15:38 -!- jungly [~jungly@host5-191-dynamic.42-79-r.retail.telecomitalia.it] has quit [Remote host closed the connection] 15:56 -!- profmac [~ProfMac@2001:470:1f0f:226:8530:7dfd:325:e852] has quit [Ping timeout: 264 seconds] 16:00 -!- leishman [~leishman@50.237.29.22] has joined #bitcoin-core-dev 16:01 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has joined #bitcoin-core-dev 16:09 -!- Dizzle [~dizzle@108.171.182.16] has quit [Ping timeout: 272 seconds] 16:16 -!- leishman [~leishman@50.237.29.22] has quit [Remote host closed the connection] 16:18 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 16:24 < luke-jr> is it preferable to de-bundle libpng from depends/qt, or patch the libpng bundled with depends/qt? 16:25 < luke-jr> cfields: ^ 16:36 -!- plankers [~plank@38.87.81.82] has quit [Ping timeout: 244 seconds] 16:48 < luke-jr> probably debundling, since Qt's copy is missing the files needed :/ 16:48 * luke-jr ponders using virtfs for gitian cache 16:51 < gmaxwell> jonasschnelli: so this protocol's use of a seperate chacha20 for the sizes is effectively halving the performance of our cypher. 16:53 < gmaxwell> jonasschnelli: the use of a message number as the nonce (instead of using a continious stream) will also decrease our performance for AVX by maybe 20%. 16:54 < sipa> gmaxwell: that's a strange gratuitous performance reduction, indeed 16:54 < sipa> i wonder why openssh did that 16:55 < gmaxwell> maybe because they thought they would read ahead lengths and thus schedule the other runs more efficiently? 16:57 < gmaxwell> https://tools.ietf.org/html/draft-josefsson-ssh-chacha20-poly1305-openssh-00#section-3 16:59 < sipa> the separation of the length and data cipher makes sense, but there's no need to discard 28 bytes from each output of K_1 17:00 < gmaxwell> right. well there is also no need to discard bytes from K_2 but it does that too. the performance hit is especially gratitious for lengths. 17:02 -!- goatpig [56eece80@gateway/web/freenode/ip.86.238.206.128] has joined #bitcoin-core-dev 17:03 -!- intcat [~zshlyk@gateway/tor-sasl/intcat] has quit [Ping timeout: 250 seconds] 17:04 -!- AaronvanW [~AaronvanW@unaffiliated/aaronvanw] has quit [Remote host closed the connection] 17:04 < gmaxwell> sipa: oh geesh, for every message we do _another_ chacha20 run to derrive the poly key. 17:05 < gmaxwell> so encrypting a single small message requires 3 runs of the chacha20 function, one to encrypt the length, one to establsh the poly1305 key, and one to encrypt the payload. 17:06 < gmaxwell> this seems pants on head stupid. 17:06 -!- intcat [~zshlyk@gateway/tor-sasl/intcat] has joined #bitcoin-core-dev 17:07 < gmaxwell> The polykey needs to be per packet for poly1305's requirements, so I suppose it's only throwing out 32 bytes of chacha output. 17:14 < gmaxwell> https://www.ietf.org/mail-archive/web/secsh/current/msg01224.html seems other people have suggested combining the poly1305 chacha block with the length encryption. 17:14 < gmaxwell> and it seems that they didn't so that you could just use a RFC5116 implementation of it. 17:14 * gmaxwell cries 17:16 -!- plankers [~plank@c-98-238-141-78.hsd1.ca.comcast.net] has joined #bitcoin-core-dev 17:24 < gmaxwell> So encrypting a 12 byte message will run on the order of 109 cycles/byte... which means that for small messages a straighforward implementation of AES-GCM would likely be faster, even on hardware without AES instructions. 17:24 < gmaxwell> (109 cycles/byte for the chacha20 part alone) 17:28 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 17:29 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 17:30 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 17:30 -!- IGHOR [~quassel@93.178.216.72] has quit [Quit: http://quassel-irc.org ? ??????????? ?????????. ????-??.] 17:31 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 17:31 < sipa> gmaxwell: but what is the average message length for us? 17:32 -!- IGHOR [~quassel@93.178.216.72] has joined #bitcoin-core-dev 17:34 < sipa> it seems we don't keep stats on message counts 17:42 -!- promag [~promag@83.223.235.81] has quit [Remote host closed the connection] 17:59 -!- meshcollider_ [uid246294@gateway/web/irccloud.com/x-gydxecdiidffgcuo] has quit [Quit: Connection closed for inactivity] 18:03 -!- profmac [~ProfMac@2001:470:1f0f:226:21c7:9b4e:db96:accb] has joined #bitcoin-core-dev 18:09 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev 18:14 -!- promag [~promag@83.223.235.81] has quit [Ping timeout: 264 seconds] 18:24 < cfields> luke-jr: hmm? 18:24 < cfields> luke-jr: if qt's copy is missing the files that need patching, what's to patch? 18:36 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds] 18:47 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 18:55 < luke-jr> cfields: libpng has optimisations for ARM and POWER in separate files missing in Qt, but Qt's copy of the normal files still tries to link them 18:57 < cfields> luke-jr: armhf/aarch64 build fine, what's different about power? 18:58 < luke-jr> cfields: I don't know how ARM works 18:58 < cfields> (not arguing, just trying to understand) 18:59 < cfields> luke-jr: anyway, breaking out libpng is fine with me. IIRC I didn't do that because it requires zlib, as does qt, so that would've meant 2 copies of zlib. But we've since broken zlib out anyway I believe. 19:00 < luke-jr> yeah 19:01 < cfields> luke-jr: while you're at it, feel free to flip -qt-jpeg to -disable-jpeg too 19:01 < cfields> something like those options, anyway 19:01 < cfields> I think we've had no need for jpegs for a long time 19:04 < luke-jr> did we ever? O.o 19:05 < cfields> pretty sure we had some at some point 19:10 < midnightmagic> ‰/w 39 19:11 < cfields> luke-jr: https://github.com/bitcoin/bitcoin/commit/f9124587ccea723dbd743e3877a7071fbb6c5732 19:12 < cfields> 0.8, heh 19:16 < gmaxwell> sipa: the most common message by far is transaction inv. 19:18 < gmaxwell> sipa: it's just so weird that it uses 3 chacha runs, the poly1305 run has 32 bytes totally unused. 19:28 -!- Krellan [~Krellan@50-242-94-241-static.hfc.comcastbusiness.net] has quit [Remote host closed the connection] 19:31 -!- davex__ [~user@97.119.150.83] has quit [Ping timeout: 264 seconds] 19:32 -!- pierre_rochard [sid299882@gateway/web/irccloud.com/x-ugovmpgchucafyyp] has joined #bitcoin-core-dev 19:41 -!- jimpo [~jimpo@ec2-13-57-39-52.us-west-1.compute.amazonaws.com] has joined #bitcoin-core-dev 19:42 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev 19:48 -!- promag [~promag@83.223.235.81] has quit [Ping timeout: 252 seconds] 20:32 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 252 seconds] 20:55 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev 20:55 -!- Rootsudo [~textual@180.190.116.243] has joined #bitcoin-core-dev 21:01 -!- Jmabsd [~jmabsd@14.0.229.89] has joined #bitcoin-core-dev 21:01 < Jmabsd> wait, so Bitcoin has the tendency to print (256 & 160bit) hashes in *reverse* order, right - block hashes, transaction hashes and merkle root hashes. 21:02 < Jmabsd> What about pubkey hashes (20B), pubkeys (32B) and signatures (64B) - are those printed in normal or reverse byte order? so, I have a P2SH pubkey script, say. in there is a 20B hash of my redeemscript, right. when I use Bitcoin Core's script disassembly function, will it print that hash in byte or normal order? i mean there is an outer extent to what Core prints in reverse order - for instance, binary transaction dumps (in hex) are in 21:02 < Jmabsd> *normal* order, not reverse. 21:06 < sipa> Jmabsd: that's just printing the bytes one by one 21:07 < sipa> it's only when a hash is interested as a number the printing gets reversed 21:07 < sipa> because the bytes are interpreted as little-endian number, but then printed in big endian for human consumption (humans want to see numbers in big endian) 21:07 < sipa> but a script is a number 21:08 < luke-jr> isn't* 21:08 < Jmabsd> gotcha. 21:08 < sipa> *indeed, isn't 21:11 < Jmabsd> aha. so let's see - if you print a hex dump of a signature (71/72/73B), that's not a hash and hence printed in normal order 21:11 < Jmabsd> a P2SH hash, for instance when printing the disassembly of a P2SH pubkey script - will the 20B hash there be printed in reverse ordeR? 21:12 < Jmabsd> also if a pubkey (32B) is printed out, could that ever be in reverse order? 21:13 < luke-jr> why don't you just try it and see? -.- 21:14 < sipa> Jmabsd: pubkeys are not 32 bytes, and they're not hashes 21:15 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Remote host closed the connection] 21:15 < Jmabsd> sipa: so the hex printer for other byte structures are never printed in reverse orders. 21:15 < sipa> indeed 21:15 < sipa> only for things that are internally treated as numbers 21:15 < Jmabsd> but.. a P2SH 20B hash, that's a hash right. for printing purposes, is it considered a hash or a byte blob? 21:15 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 21:15 < sipa> nope! 21:15 < sipa> because the printer cannot know it is a hash 21:16 < sipa> you'd need to execute the script to know it is treated as such 21:16 < sipa> the script opcode is just "put some bytes on the stack" 21:17 < sipa> so, not reversed there 21:20 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [Ping timeout: 264 seconds] 21:20 -!- Jmabsd [~jmabsd@14.0.229.89] has quit [Ping timeout: 268 seconds] 21:21 -!- Jmabsd [~jmabsd@14.0.171.114] has joined #bitcoin-core-dev 21:21 < Jmabsd> (sorry disconnect) 21:21 < Jmabsd> last, > interesting. except for the HD wallet root seed (160b=20B), there is no instance ever where a 20B hash e.g. in P2SH pubkeyscript, is printed in reverse order. 21:21 < Jmabsd> > sipa, right and when getting a disassembly printout in Bitcoin Core and related tools, those 20B:s are printed in normal order 21:28 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has joined #bitcoin-core-dev 21:32 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has quit [Remote host closed the connection] 21:33 -!- d9b4bef9 [~d9b4bef9@web501.webfaction.com] has joined #bitcoin-core-dev 21:40 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 268 seconds] 22:36 -!- promag [~promag@83.223.235.81] has joined #bitcoin-core-dev 22:41 -!- promag [~promag@83.223.235.81] has quit [Ping timeout: 268 seconds] 22:49 -!- vexbuy [~vexbuy@147.red-83-47-212.dynamicip.rima-tde.net] has quit [] 23:07 -!- vexbuy [~vexbuy@89.39.107.192] has joined #bitcoin-core-dev 23:16 < Jmabsd> the proper way to phrase Core's reversing policy is something like, "any hash that is not part of another binary blob or produced as script data, is hex-serialized in reverse byte order." 23:17 < Jmabsd> i'd hope any hash values introduced in the future will not be reversed though. 23:26 < sipa> i don't see why not 23:26 < sipa> we've always treated hash outputs as numbers and printed them as such 23:26 < sipa> if byte swapping is the hardest problem to deal with, i'm not very worried :) 23:43 -!- Jmabsd [~jmabsd@14.0.171.114] has quit [Ping timeout: 264 seconds] 23:53 -!- vexbuy [~vexbuy@89.39.107.192] has quit [Remote host closed the connection] 23:54 -!- vexbuy [~vexbuy@89.39.107.192] has joined #bitcoin-core-dev 23:58 -!- vexbuy [~vexbuy@89.39.107.192] has quit [Ping timeout: 252 seconds] --- Log closed Sat Aug 25 00:00:33 2018