--- Day changed Sat Jul 09 2016
00:00 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
00:01 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
01:47 -!- Thapgich [~tesla@212.55.95.116] has joined #bitcoin-core-dev
02:03 -!- Giszmo [~leo@0.red-95-120-58.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
02:13 -!- afk11 [~afk11@unaffiliated/afk11] has quit [Ping timeout: 252 seconds]
02:22 -!- afk11 [~afk11@109.255.154.81] has joined #bitcoin-core-dev
02:22 -!- afk11 [~afk11@109.255.154.81] has quit [Changing host]
02:22 -!- afk11 [~afk11@unaffiliated/afk11] has joined #bitcoin-core-dev
02:30 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
02:33 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
02:36 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 250 seconds]
02:36 -!- nanotube [~nanotube@unaffiliated/nanotube] has quit [Ping timeout: 244 seconds]
02:37 -!- afk11 [~afk11@unaffiliated/afk11] has quit [Ping timeout: 258 seconds]
02:37 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-zhxocdvqoolehzga] has joined #bitcoin-core-dev
02:37 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
02:43 -!- afk11 [~afk11@109.255.154.81] has joined #bitcoin-core-dev
02:43 -!- afk11 [~afk11@109.255.154.81] has quit [Changing host]
02:43 -!- afk11 [~afk11@unaffiliated/afk11] has joined #bitcoin-core-dev
02:50 -!- nanotube [~nanotube@unaffiliated/nanotube] has joined #bitcoin-core-dev
03:02 -!- achow101 [~achow101@pool-108-2-58-173.phlapa.fios.verizon.net] has quit [Ping timeout: 272 seconds]
03:15 < GitHub108> [bitcoin] jonasschnelli opened pull request #8323: Add HD keypath to CKeyMetadata, report metadata in validateaddress (master...2016/07/hd_013) https://github.com/bitcoin/bitcoin/pull/8323
03:15 < jonasschnelli> I think we should consider #8323 (+37 −1) for 0.13 
03:16 < GitHub29> [bitcoin] jonasschnelli closed pull request #8205: [Wallet] add HD keypath to CKeyMetadata, report over validateaddress (master...2016/06/hd_metadata) https://github.com/bitcoin/bitcoin/pull/8205
03:18 -!- achow101 [~achow101@pool-108-2-58-173.phlapa.fios.verizon.net] has joined #bitcoin-core-dev
03:29 -!- Giszmo1 [~leo@112.red-79-153-18.dynamicip.rima-tde.net] has joined #bitcoin-core-dev
03:31 -!- Giszmo [~leo@0.red-95-120-58.dynamicip.rima-tde.net] has quit [Ping timeout: 258 seconds]
04:03 < GitHub121> [bitcoin] jonasschnelli opened pull request #8324: [Wallet] keep HD seed during salvagewallet (master...2016/07/hd_salvage) https://github.com/bitcoin/bitcoin/pull/8324
04:04 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev
04:05 -!- Giszmo1 [~leo@112.red-79-153-18.dynamicip.rima-tde.net] has quit [Ping timeout: 252 seconds]
04:05 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Client Quit]
04:10 -!- belcher [~user@unaffiliated/belcher] has joined #bitcoin-core-dev
04:13 -!- [Author] [~Author]@2401:a400:3202:2000:bad:d09:15:90d] has joined #bitcoin-core-dev
04:28 -!- achow101 [~achow101@pool-108-2-58-173.phlapa.fios.verizon.net] has quit [Ping timeout: 276 seconds]
04:30 -!- JackH [~Jack@79-73-186-51.dynamic.dsl.as9105.com] has joined #bitcoin-core-dev
04:44 -!- achow101 [~achow101@pool-108-2-58-173.phlapa.fios.verizon.net] has joined #bitcoin-core-dev
05:41 -!- cryptapus_ [~cryptapus@unaffiliated/cryptapus] has joined #bitcoin-core-dev
05:49 -!- cryptapus_ [~cryptapus@unaffiliated/cryptapus] has quit [Ping timeout: 276 seconds]
06:04 -!- arubi_ [~ese168@unaffiliated/arubi] has joined #bitcoin-core-dev
06:07 -!- arubi [~ese168@unaffiliated/arubi] has quit [Ping timeout: 276 seconds]
06:14 -!- arubi_ is now known as arubi
06:27 -!- arubi [~ese168@unaffiliated/arubi] has quit [Quit: Leaving]
06:28 -!- arubi [~ese168@unaffiliated/arubi] has joined #bitcoin-core-dev
06:46 -!- fifth__ [~fifth@150.16.90.92.rev.sfr.net] has joined #bitcoin-core-dev
06:56 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
07:05 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has joined #bitcoin-core-dev
07:12 -!- fifth__ [~fifth@150.16.90.92.rev.sfr.net] has quit [Remote host closed the connection]
07:26 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 240 seconds]
07:29 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
07:38 -!- davidlj95 [~davidlj95@deic-dyn-232.uab.es] has quit [Remote host closed the connection]
07:39 -!- dgenr8 [~dgenr8@unaffiliated/dgenr8] has joined #bitcoin-core-dev
07:47 < GitHub68> [bitcoin] Pranit-Harekar opened pull request #8325: 0.12 (master...0.12) https://github.com/bitcoin/bitcoin/pull/8325
07:59 < GitHub44> [bitcoin] MarcoFalke closed pull request #8325: 0.12 (master...0.12) https://github.com/bitcoin/bitcoin/pull/8325
08:10 -!- catlas [d0a7fee4@gateway/web/freenode/ip.208.167.254.228] has joined #bitcoin-core-dev
08:32 -!- kadoban [~mud@unaffiliated/kadoban] has joined #bitcoin-core-dev
08:39 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 276 seconds]
08:49 < btcdrak> Happy Halving Countdown https://www.youtube.com/watch?v=ovXOZjxFB00
08:54 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
09:47 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has joined #bitcoin-core-dev
09:48 -!- PatBoy [xyz@192.99.249.215] has quit [Quit: peace]
09:49 -!- PatBoy [xyz@192.99.249.194] has joined #bitcoin-core-dev
09:51 -!- jtimon [~quassel@55.31.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev
10:22 -!- MarcoFalke [~marco@host10-2.natpool.mwn.de] has left #bitcoin-core-dev []
10:36 -!- jgarzik [~jgarzik@unaffiliated/jgarzik] has joined #bitcoin-core-dev
10:46 -!- jtimon [~quassel@55.31.134.37.dynamic.jazztel.es] has quit [Ping timeout: 264 seconds]
11:13 -!- catlas [d0a7fee4@gateway/web/freenode/ip.208.167.254.228] has left #bitcoin-core-dev []
11:27 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
11:30 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 244 seconds]
12:00 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 244 seconds]
12:17 -!- jonath___ [~jonathan@c-98-210-128-59.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
12:22 -!- jonatha__ [~jonathan@c-98-210-128-59.hsd1.ca.comcast.net] has joined #bitcoin-core-dev
12:22 -!- jonath___ [~jonathan@c-98-210-128-59.hsd1.ca.comcast.net] has quit [Read error: Connection reset by peer]
12:23 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
12:25 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 240 seconds]
12:27 -!- jonatha__ [~jonathan@c-98-210-128-59.hsd1.ca.comcast.net] has quit []
12:44 -!- slackircbridge [~slackircb@45.55.41.36] has quit [Remote host closed the connection]
12:45 -!- slackircbridge [~slackircb@45.55.41.36] has joined #bitcoin-core-dev
13:23 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
13:26 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 250 seconds]
13:38 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
13:38 -!- Thapgich [~tesla@212.55.95.116] has quit [Quit: Leaving]
13:55 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
13:57 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 252 seconds]
13:59 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
14:00 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 246 seconds]
14:20 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 276 seconds]
14:28 -!- Guyver2 [~Guyver2@guyver2.xs4all.nl] has quit [Remote host closed the connection]
14:37 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
14:48 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 272 seconds]
14:52 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
15:31 -!- instagibbs [~instagibb@pool-100-15-114-5.washdc.fios.verizon.net] has quit [Ping timeout: 246 seconds]
16:10 -!- laurentmt [~Thunderbi@176.158.157.202] has joined #bitcoin-core-dev
16:13 -!- laurentmt [~Thunderbi@176.158.157.202] has quit [Client Quit]
16:18 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 246 seconds]
16:18 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
16:48 < sipa> http://bitcoin.sipa.be/utxo_size.png
16:55 < petertodd> sipa: what's the red represent/
16:55 < petertodd> ?
16:55 < petertodd> sipa: oh, under 1uBTC?
16:55 < sipa> yes
16:55 < petertodd> sipa: cool, thanks! mind if I link that from the blog post I'm writing?
16:55 < sipa> it is "what would the utxo set size be if we deleted all outputs below value X"
16:56 < petertodd> sipa: (as in, I'll make a copy of it today, and then link w/ a URL for the live version)
16:57 < sipa> the graph is not being automatically updated (yet)
16:57 < sipa> so feel free
16:57 < petertodd> sipa: thanks!
16:58 < sipa> i guess there is possible ambiguity: every vertical crosscut of the graph represents the entire UTXO set as it was at the date shown by the X axis
16:58 < sipa> it's not a breakdown of the current UTXO set by when the entries were created
16:59 < gmaxwell> I just suggested sipa create a breakdown one a little bit ago.
17:00 < petertodd> I could make use of a breakdown by age fwiw
17:01 < sipa> yeah, i'll create such a graph too
17:01 < sipa> should be easier as it doesn't require reindexing
17:01 < gmaxwell> I'm thinking it would be interesting to solve the following optimization problem:  given a set of utxo size, birth/death times, value.  Find the coefficients of multinomial a f(value,size,age_in_blocks) such that a fixed cache of size N with retention sorted by f() has the highest hitrate (item still in cache at utxo death).
17:02 < petertodd> sipa: thanks!
17:03 < petertodd> gmaxwell: though note that you probably can't bake something like that into a consensus thing like txo commitments w/ utxo cache, as it changes peoples' behavior...
17:03 < petertodd> gmaxwell: potentially still useful for local optimisation though
17:08 < gmaxwell> petertodd: well you can to some extent, for example, having a different cache policy at some threshold, like 1BTC... 21 million txo is the absolute maximum number of outputs at 1BTC+ so there is a pretty strong upper bound on whatever impact you could have on encouraging people to make utxo at or over 1 BTC. :)
17:10 < petertodd> gmaxwell: I mean, if you take the coefficients from prior data, they're likely to be wrong after people change their behavior - if you use coefficients, you need to have a different rational is all I'm saying
17:10 < petertodd> gmaxwell: keeping higher value UTXO's in cache longer probably does make sense
17:11 < gmaxwell> yea sure.. the full answer isn't probably that useful as a consensus rule
17:11 -!- johhnyfive [6bbf2418@gateway/web/freenode/ip.107.191.36.24] has joined #bitcoin-core-dev
17:11 < gmaxwell> The part of the answer that tells you some value breakpoint(s) may be.
17:12 < gmaxwell> because even if they're slightly wrong due to changing usage which they themselves incentivize, the whole prospect of having value relative retention is reasonable. 
17:13 < gmaxwell> e.g. if age * value is a good scoring function on the past data, it probably is a robust one, which could be prudently used in consensus.
17:14 < petertodd> yup
17:15 < gmaxwell> or some polynomial on age * value.  Or really I think any degree multinomial on age and value is probably also okay.  Only size is one that is probably busted. :)
17:15 < petertodd> gmaxwell: speaking of, what approaches are good for writing polynomials in consensus critical code? (I wonder if someone has a writeup on that)
17:17 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has joined #bitcoin-core-dev
17:20 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 272 seconds]
17:25 < gmaxwell> So-- if possible, probably best by converting it to a simple segmented linear approximation. But failing that, I would assume fixed point with horner's method (see wikipedia) which is more computationally efficient and has better numerical properties than the naieve way you'd compute it.
17:27 < gmaxwell> this is where you write is as a c0 + x * (c1 + x * (c2 + x *....
17:28 < gmaxwell> petertodd: there are all sorts of 'consensus critical' polynomials in opus (ones where a discrepency in the integer value returned causes a total entropy decoding failure)-- they never were a big issue, they're written like that, and we tested them exhaustively. no biggie.
17:31 < petertodd> gmaxwell: cool, thanks. re: exhaustively, I assume that's 16bit ints at most?
17:31 < gmaxwell> no, 32 bits.
17:32 < petertodd> gmaxwell: huh, how does that work?
17:32 < gmaxwell> for some function thats just doing some multiplies and adds, trying all 32 bit inputs isn't a big deal.
17:32 < petertodd> gmaxwell: wait, as in 2^64?
17:33 < sipa> i assume it's a function with 1 input :)
17:33 < kanzure> https://github.com/xiph/opus/blob/58dbcf23f3aecfb9c06abaef590d01bb3dba7a5a/celt/cwrs.c#L164
17:33 < gmaxwell> no, as in computing f(x) where x is some 32 bit value and x is a single variable polynomial in x.  :)
17:34 < petertodd> gmaxwell: yeah, that makes a lot more sense :)
17:34 < petertodd> gmaxwell: I'm writing a spec for dex, and it's interesting how you can make an argument for only supporting 16-bit ints natively, as you can exhaustively test them
17:34 < gmaxwell> yes, indeed that is a bit normative polynomial, though I think the current CWRS code there mostly uses tables.
17:36 < gmaxwell> petertodd: thats something you could argue, though it would have to be weighed against footgun and bloat risks.
17:36 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has joined #bitcoin-core-dev
17:36 < gmaxwell> (though I suppose I could make a anti-footgun argument-- the user is much more likely to _know_ the range of the type is limited, when it's so limited they are constantly forced to deal with it)
17:37 < gmaxwell> over and underflow can be defined as script failure.
17:37 < petertodd> gmaxwell: yeah, it'd only be practical if you can write reasonably efficient n-bit add/multiply/etc. routines, and make them part of the built-in library
17:38 < petertodd> gmaxwell: yes, I'm planning on under/overflow to be a failure condition (probably with wrapping and/or overflow variants)
17:38 < gmaxwell> petertodd: also a commonly interesting case is things like where one input has small range, and that doesn't really impede exaustive testing.
17:38 < gmaxwell> e.g. U32 * U8.  It's really common in software to take arbritary numbers and multiply or divide them by small constants.
17:38 < petertodd> gmaxwell: so, what I noted in my draft spec doc, is that interestingly economicly relevant numbers often have quite large ranges: e.g. coin value
17:39 < gmaxwell> like computing 2/3 of a coin value. 
17:39 < petertodd> gmaxwell: though, consensus critical economic use-cases just need summation, not multiplication/division (usually)
17:40 < sipa> petertodd: but but demurrage
17:40 < gmaxwell> for example, you may want to compute input * 2/3, and input - input * 2/3.  
17:40 < gmaxwell> for value splits in a contract.
17:40 < gmaxwell> and input is 64 bits.
17:41 < petertodd> sipa: ok, austrian economics... :P
17:41 -!- jtimon [~quassel@55.31.134.37.dynamic.jazztel.es] has joined #bitcoin-core-dev
17:42 < petertodd> gmaxwell: well, once you talk about contracts more genreally, you get interest calculations, which gets very complex...
17:43 < gmaxwell> polynomial approximations of interest calculations generally work fine over limited ranges.
17:43 < petertodd> gmaxwell: but I'm assuming sane contracts would generally only need a few calculations along those lines, so slower approaches should be ok
17:43 < petertodd> yeah, polynomial approx being one approach
17:44 < petertodd> in any case, it's looking like having reasonable type checking is a way harder and more complex problem :)
17:45 < gmaxwell> one nice way for exponential functions is to use CLZ to get the integer part of a base 2 log, and then use a polynomial correction.
17:45 < petertodd> gmaxwell: CLZ?
17:45 < sipa> count leading zeroes
17:45 < petertodd> sipa: ah!
17:45 < sipa> __builtin_clz
17:49 < gmaxwell> er actually its the log you want the clz for, for exp you just need to use the leading bits of the number. 
17:49 < gmaxwell> https://github.com/xiph/opus/blob/58dbcf23f3aecfb9c06abaef590d01bb3dba7a5a/celt/mathops.h#L184 is such an example. (of course to get to any other base for the exponential is just some scaling)
17:50 -!- belcher [~user@unaffiliated/belcher] has quit [Quit: Leaving]
17:50 < petertodd> gmaxwell: a neat article to write would be numerical methods for consensus critical code :)
17:51 < sipa> "Bounded memory, bounded CPU usage... and bounded errors"
17:51 < gmaxwell> I think they mostly end up like numerical methods for fixed point realtime dsp.
17:51 < petertodd> sipa: lol, did chris send you a copy of my spec? :P
17:51 < sipa> no
17:51 < sipa> well, maybe he did, but i did not look at it
17:51 < petertodd> sipa: I mean, I have basically the exact same sentence in it - though no surprise, same problem :)
17:52 < petertodd> gmaxwell: yeah, probably true
17:52 < gmaxwell> with perhaps some additional considerations for exhaustive testing and/or formal verification.
17:53 < gmaxwell> but yea, the other reason you exaustively test these approximations is to characterize the worst case error: https://github.com/xiph/opus/blob/58dbcf23f3aecfb9c06abaef590d01bb3dba7a5a/celt/mathops.c#L202
17:54 < petertodd> bbl, need a bigger battery...
17:55 < midnightmagic> i wonder how heavy that special 9-cell thinkpad thing is
18:09 < petertodd> midnightmagic: doesn't help that mine is a few years old - batteries wearing out
18:09 < sipa> petertodd: protip, when buying a new laptop, buy an extra battery that fits... once your battery wears out they'll be harder to find
18:10 < petertodd> sipa: good advice - though I've never actualy bought a laptop new
18:10  * midnightmagic has no user-replaceable battery in his mbp. :(
18:10 < midnightmagic> petertodd: why not?
18:10 < petertodd> midnightmagic: because I act like I'm a poor student :P
18:12 < petertodd> midnightmagic: I have a t520 right now, which I think is about four years old
18:18 < midnightmagic> Humility and reasonable resource management are virtues.
18:20 < petertodd> midnightmagic: well, I was going through my corporate expenses the other day... and a new laptop would be a drop in the bucket compared to what gets spent on me travelling
18:21 < midnightmagic> i hate travelling :( the world is made for people much, much smaller than I am.
18:22 < petertodd> midnightmagic: I actually find I get more work done; I'm no tourist so when I'm in the middle of a foreign country I tend to find somewhere to sit down with my laptop :)
18:25 < midnightmagic> i do too, all the stuff that is on the to-do list in my home can't be addressed so I can let it go
18:25 -!- Ylbam [uid99779@gateway/web/irccloud.com/x-zhxocdvqoolehzga] has quit [Quit: Connection closed for inactivity]
18:26 < petertodd> midnightmagic: yeah, and when I'm home, friends want to like, hang out with me and take up all my time :P
18:30 < midnightmagic> psh. friends. so annoying.
18:56 < petertodd> sure
18:56 < petertodd> er, wrong window - stupid lag
19:02 < sipa> haha
19:12 < gmaxwell> sipa: lithion ion batteries of varrious type have a shelf life, sadly.
19:12 < gmaxwell> The unused one will also fade.
19:13 < gmaxwell> But fortunately if you buy something like a thinkpad, people make batteries for them forever.
19:13 < gmaxwell> You can buy batteries for ten year old thinkpads no problem.
19:20 < gmaxwell> petertodd: go stab the people your blog post has confused: https://www.reddit.com/r/Bitcoin/comments/4s3a9r/segwit_code_review_update/
19:41 < kanzure> consensus-related non-malleability vs wallet/p2p-level, is my guess.
19:42 < petertodd> kanzure: it's just a flaw in the way the mempool/p2p refers to segwit txs; has little if anything to do with consensus
19:46 -!- molz [~molly@unaffiliated/molly] has quit [Ping timeout: 244 seconds]
19:46 < petertodd> kanzure: tl;dr: is that the p2p asks for txs by txid, which doesn't commit to the witness, and marks invalid txs by txid, without being able to consistently know if a tx was invalid due to a bad witness
19:46 < petertodd> equaly, s/invalid/unacceptable due to fee, size, whatever/
19:46 < sipa> i actually don't think the flaw is referring by txid instead of wtxid
19:47 < petertodd> sipa: how so?
19:47 < sipa> but we should have made resource limits part of the base transaction, not the witness
19:47 < kanzure> yes i was wondering about things like "how to actually show a node that a certain tx is valid later, if at first it receives a bad witness" :\
19:47 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
19:48 < petertodd> sipa: what do you mean by that?
19:48 < sipa> (fees, sigop count, byte sizes) go in scriptSig... or even better, in the inv
19:48 < sipa> so a node can decide to not process before you know... processing
19:48 < petertodd> sipa: duplicating that stuff has historically lead to endless problems, basically because you have to check it twice
19:49 < sipa> how do you mean?
19:49 < petertodd> sipa: in a decent system, processing even invalid txs is something that happens very quickly, so there's no DoS attack
19:49 < petertodd> sipa: notice how satoshi screwed up sigops from the beginning...
19:50 < sipa> unfortunately, fees and sigop counting require fetching the utxos
19:50 < petertodd> sipa: in any of these schemes, you still have to count up sigops as they're actually executed, to check that the sum matches (or is less than) the claimed sum
19:50 < sipa> of course
19:50 < petertodd> sipa: so? fetching utxos can't be expensive, or we've already lost
19:50 < sipa> but a mismatch can then actually result in a ban, because it cannot happen accidentally
19:51 < petertodd> sipa: if we used wtxids, you could still ban based on that
19:51 < sipa> but our rate limiting is based on feerate, which depends on fee, which we cannot enforce until we've done the effort
19:51 -!- johhnyfive [6bbf2418@gateway/web/freenode/ip.107.191.36.24] has quit [Ping timeout: 250 seconds]
19:52 < sipa> if there is no rate limit, even a cheap validation per transaction will be too much
19:52 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Remote host closed the connection]
19:52 < petertodd> sipa: huh? someone sends you a DoS tx, just ban them - there's no reason *legit* transactions should take significant cost to accept
19:54 < sipa> petertodd: so there is a trivial solution... fully validate every transaction you asked for
19:54 < sipa> so you don't prematurely discard a transaction before finding out it was an attempted attack
19:55 < sipa> it is too expensive, or invalid, or malleated... you can ban who sent it
19:55 < petertodd> sipa: I think you're missing my point: the threshold that we consider it an "attempted attack" should be low enough that there's no DoS issues; txs fundementally should never be expensive to validate, and cases where they are should be non-standard, and eventually, removed entirely from the protocol
19:56 < sipa> yes
19:56 < sipa> i agree
19:57 < sipa> but the issue here is that we fail to detect whether a too expensive transacrion is due to its creator or due to who relayed it
19:57 < petertodd> sipa: right, but wtxid solves that issue
19:58 < petertodd> if relayer malleates, we'll still ask for a different version of the same txid if another peer gives us it
19:58 < sipa> yes, it would... but it adds complications
19:58 < petertodd> how so?
19:58 < sipa> you need indexes by both txid and wtxid..
19:58 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #bitcoin-core-dev
19:58 < sipa> and you always risk requesting things twice
19:58 < petertodd> sipa: in what, the mempool/p2p?
19:59 < petertodd> but they are different things!
19:59 < sipa> in my view they are the same thing
19:59 < sipa> one with an optional part omitted
19:59 < sipa> except we only find out too late that it was not optional
20:00 < petertodd> well, I don't agree at all - the optional part has a non-optinal effect on the tx
20:00 < sipa> for those who care (which a full node does)
20:00 < sipa> that's a semantic discussion, though
20:00 < sipa> i can see your point
20:00 < petertodd> I certainly care: my tx fees depend on the witness
20:01 -!- frankenmint [~frankenmi@174-25-9-165.ptld.qwest.net] has joined #bitcoin-core-dev
20:01 < petertodd> I may also have a protocol where I'm publishing something in the witness, e.g. hashlock
20:01 < gmaxwell> Cleanstack means that ordinary transactions cannot have their fees increased without invalidating them. (If not for that I would have already recommended we have some preprocessing to strip transactions as they're relayed)
20:02 < sipa> i think the easiest solution is to validate scripts even for things we know we won't accept
20:02 < gmaxwell> You should probably assume that relaying nodes will perform witness stripping in the future.
20:02 < sipa> we have spent almost all the work anyway (fetched inputs, and wasted bandwidth)
20:02 < petertodd> gmaxwell: with currnet tx patterns yes; it's non-trivial to make txs with more complex scripts that have non-malleable witnesses
20:03 < gmaxwell> (by witness stripping I mean compacting the witness to the minimal data needed to be valid, as best it can determine)
20:03 < petertodd> sipa: as in, valiate scripts to determine if the witness is wrong?
20:03 < gmaxwell> Yes.
20:03 < gmaxwell> This was something I had been advocating for a while because there are some other potential DOS attacks that exist because we don't.
20:03 < gmaxwell> (a while = before segwit existed)
20:04 < petertodd> well, again, that assumes you know how to clean witnesses - there are tx patterns where that's not trivial (and indeed, the user may intentionally have more than one form of witness for the same txid)
20:04 < gmaxwell> though without an enforced feefilter its a bit less than ideal.
20:04 < gmaxwell> petertodd: sure any cleaning would always be a best effort attempt.
20:05 < petertodd> I'm still of the opinion that asking for a wtxid is a way simpler overall conceptual design (obvs implementation level may suck due to historical baggage)
20:05 < sipa> this is orthogonal to fetching by wtxid, of course
20:05 < petertodd> it's not orthogonal at all: if I manage to clean a script significantly, I want my peers to get it, and then say "hey, this is way smaller, lets replace it in our mempool..."
20:05 < gmaxwell> yea, I agree it's orthorgonal, fetching by wtxid has an amplification attack vector that is kind of sad.
20:06 < petertodd> gmaxwell: what's the vector?
20:06 < sipa> gmaxwell: presenting multiple versions of the same transaction with different witness?
20:06 < gmaxwell> The amplification attack vector is that I create grab transactions and relay witness malleations to every node in the network, different version to each node, so when I happen to get txn early every node ends up with a different txid and you get N^2 bandwidth forwarding it around.
20:07 < petertodd> gmaxwell: but you can do that anyway with your own txs
20:09 < sipa> you could of course create invs with both txids and wtxids
20:09 < petertodd> sipa: yeah, that'd be fine
20:09 < petertodd> sipa: and obviously, our peer can tell us it's segwit and wants us to do that
20:10 < sipa> except that also breaks your try to replace with smaller witness use case
20:10 < petertodd> sipa: why? once you go down that rabbit hole, you can also advertise length
20:11 < sipa> petertodd: yes, that was my suggestion in the beginning of the discussion :)
20:11 < sipa> advertize sigops/size/fees in the inv
20:11 < petertodd> sipa: no, you suggested having the tx _commit_ to that info, which is a very different thing; non-consensus critical code advertising length isn't a big deal
20:11 -!- kekstone [~dn@c-73-14-252-164.hsd1.co.comcast.net] has joined #bitcoin-core-dev
20:12 < sipa> petertodd: reread my sentence
20:12 < petertodd> sipa: ah, ok, I have no issues with that
20:12 < petertodd> sipa: (I've been thinking about this exact kind of issue for my dex specification actually)
20:12 < sipa> "... or even better, in the inv$
20:12 < petertodd> sipa: yeah, for mempool I think invs advertising this stuff makes a lot of sense
20:13 < petertodd> for starters, if we screw that up, it's relatively easy to fix :)
20:13 < sipa> though i think it's not necessarily an urgent issue
20:14 < sipa> the worst case is that a bad peer can poison your reject cache, preventing you from getting a legitimate transaction before it confirms
20:14 < petertodd> so, is a reasonable game plan to releast segwit with the current p2p design, and then add wtixds to invs later? (potentially with sigops/size in the inv)
20:14 < petertodd> sipa: it's a good thing no-one relies on zeroconf anymore :P
20:14 < sipa> i'm sure there are other ways that you can accomplish that
20:14 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Remote host closed the connection]
20:14 < sipa> (like inving and not responding to getdata)
20:14 < gmaxwell> wrt different kinds of relaying later, mostly I thought those improvements would go into mempool sync.
20:15 < gmaxwell> and rather than wtxid relay,  wtxid,feerate,txid tuples (probably truncated/quantized) may make more sense. 
20:16 < sipa> yeah, for mempool sync we can redesign things cleanyl
20:16 < petertodd> gmaxwell: note too how schemes like txo commitments allow - and expect - nodes to do significant modifications to txs
20:16 < gmaxwell> (or even wtxid, feerate, vin id with lowest hash)
20:18 < gmaxwell> in any case, optimal sync behavior in the presence of double spends (of any kind) isn't a nut I've cracked yet.
20:18 < sipa> petertodd: yes, the priority should be to make sure no internal inconsistency or banning of unaware nodes occur
20:18 < gmaxwell> I think I have constructions for schemes which are close to optimal absent doublespends.
20:18 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #bitcoin-core-dev
20:18 < gmaxwell> we already improved relay efficiency a LOT in 0.13, fwiw.
20:19 < sipa> petertodd: rejectioncache behaviour can either degenerate into attackers preventing you from receiving transactions on the one hand
20:19 < sipa> or to the old pre-rejectioncache bevahiour of requesting failed txn from every peer (which is made much less bad due to feefilter already)
20:21 < gmaxwell> there are already several trivially exploited ways where an attacker can massively delay you getting a transaction.
20:21 < petertodd> sipa: well, again, an attacker can do that DoS attack without segwit by just sending multiple slightly different versions of the same tx
20:22 < gmaxwell> (e.g. just inv the damn thing from many sockets and don't respond.
20:22 < gmaxwell> )
20:24 < sipa> yeah
20:30 -!- frankenmint [~frankenmi@174-25-9-165.ptld.qwest.net] has quit [Remote host closed the connection]
20:51 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has quit [Ping timeout: 250 seconds]
20:52 -!- spudowiar [~spudowiar@unaffiliated/spudowiar] has quit [Quit: Leaving.]
20:59 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
21:00 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
21:03 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 250 seconds]
21:03 -!- frankenmint [~frankenmi@174-25-9-165.ptld.qwest.net] has joined #bitcoin-core-dev
21:08 -!- moli [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
21:10 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
21:11 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
21:14 -!- frankenmint [~frankenmi@174-25-9-165.ptld.qwest.net] has quit [Remote host closed the connection]
21:21 -!- molz [~molly@unaffiliated/molly] has joined #bitcoin-core-dev
21:23 -!- moli [~molly@unaffiliated/molly] has quit [Ping timeout: 250 seconds]
21:26 -!- gribble [~gribble@unaffiliated/nanotube/bot/gribble] has joined #bitcoin-core-dev
21:41 -!- cryptapus_ [~cryptapus@199.47.67.56] has joined #bitcoin-core-dev
21:41 -!- cryptapus_ [~cryptapus@199.47.67.56] has quit [Changing host]
21:41 -!- cryptapus_ [~cryptapus@unaffiliated/cryptapus] has joined #bitcoin-core-dev
21:58 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
21:59 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
21:59 -!- Netsplit *.net <-> *.split quits: wangchun, teward, Michail1, [Author], Justinus, harding, crudel, justanotheruser, owowo, GreenIsMyPepper,  (+18 more, use /NETSPLIT to show all of them)
22:01 -!- adamg [~akg@50.242.93.33] has quit [Ping timeout: 272 seconds]
22:20 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
22:21 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
22:22 -!- Chris_Stewart_5 [~Chris_Ste@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 244 seconds]
22:36 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
22:37 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
22:54 -!- Michail1 [~michail@michail.com] has joined #bitcoin-core-dev
23:11 -!- cryptapus_ [~cryptapus@unaffiliated/cryptapus] has quit [Ping timeout: 264 seconds]
23:22 -!- Alopex [~bitcoin@cyber.dealing.ninja] has quit [Remote host closed the connection]
23:22 -!- adamg [~akg@50.242.93.33] has joined #bitcoin-core-dev
23:35 -!- arubi [~ese168@unaffiliated/arubi] has quit [Ping timeout: 272 seconds]
23:44 -!- arubi [~ese168@unaffiliated/arubi] has joined #bitcoin-core-dev
23:45 -!- Alopex [~bitcoin@cyber.dealing.ninja] has joined #bitcoin-core-dev
23:49 -!- arubi [~ese168@unaffiliated/arubi] has quit [Ping timeout: 252 seconds]