--- Log opened Mon Nov 26 00:00:34 2018
00:10 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-jnsqzbnguqjzylgl] has joined #rust-bitcoin
01:17 -!- simian_za_ [sid314183@gateway/web/irccloud.com/x-dyvkdmsnzdlvvxal] has joined #rust-bitcoin
01:25 -!- Netsplit *.net <-> *.split quits: simian_za, dpc
01:25 -!- simian_za_ is now known as simian_za
01:31 -!- dpc [dpcmatrixo@gateway/shell/matrix.org/x-vfcjsnvrobtshpds] has joined #rust-bitcoin
03:06 < stevenroose> I used to work on btcd (Go full node) from time to time, and suddenly we seems swarmed by some bcashers that started the gcash project to create bchd.  They are fixing issue that have been open for years etc.
03:07 < stevenroose> Usually not up to the code quality standards that btcd requires, though, but nonetheless there seems to be more than just fork works going on there :D
03:07 < stevenroose> fork wars*
04:50 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #rust-bitcoin
05:21 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection]
07:10 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #rust-bitcoin
07:58 < BlueMatt> ariard: alright, will hold off
08:09 -!- TamasBlummer1 [~Thunderbi@p200300DD67286B96C09D22D3048757B8.dip0.t-ipconnect.de] has joined #rust-bitcoin
08:11 -!- TamasBlummer [~Thunderbi@p200300DD67286B43C09D22D3048757B8.dip0.t-ipconnect.de] has quit [Ping timeout: 264 seconds]
08:11 -!- TamasBlummer1 is now known as TamasBlummer
08:16 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection]
10:39 -!- elichai2 [uid212594@gateway/web/irccloud.com/x-jnsqzbnguqjzylgl] has quit [Quit: Connection closed for inactivity]
10:54 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #rust-bitcoin
11:24 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has quit [Remote host closed the connection]
11:25 -!- sipa [~pw@gateway/tor-sasl/sipa1024] has joined #rust-bitcoin
12:42 < stevenroose> something going down in JS land: https://np.reddit.com/r/Bitcoin/comments/a0m39e/copay_wallet_application_targeted_in_hack/
12:42 < stevenroose> Some guy that created a popular package handed ownership to a random guy and he injected some random piece of code that seems to specifically target the copay wallet and some fork of it.
12:52 < gmaxwell> so sad that rust ecosystem has essentially the same security model as the nodejs one...
13:24 < sgeisler> @gmaxwell: Except that you should pin your dependencies for applications and in theory you should review them on upgrades. I know the last part happens like ... never :/
13:24 < sgeisler> but what would a good solution to that look like?
13:25 < gmaxwell> and if you _do_ attempt to do that, everything breaks in any case.
13:25 < sgeisler> Have a committee that does audits and approves certain versions of popular crates?
13:25 < sgeisler> what do you mean with "breaks"?
13:26 < gmaxwell> interop problems with other packages, interop problems with compiler updates.
13:26 < gwillen> gmaxwell: just to check, rust is not directly implicated anywhere here, right?
13:26 < gwillen> it's just that theoretically one could do the same sort of attack against a rust package?
13:27 < gmaxwell> gwillen: right. this is an npm package.
13:27 < sipa> i assume there is a bit of cultural difference though
13:27 < gwillen> yeah, a bit
13:27 < gwillen> npm has long been known to be a dumpster fire
13:27 < gwillen> not sure how much of that is purely cultural
13:28 < sgeisler> I agree, if you update dependencies you may have to upgrade sub-dependencies. But once you have a cargo.lock it should always work on stable.
13:29 < gmaxwell> gwillen: the same deep depencencies and unreviewed changes stuff happens with rust. I don't see how it's substantially different, except the magnitude perhaps.
13:30 < gwillen> well, I'd be curious to see what the dependency fanout looks like on npm vs rust
13:30 < gwillen> I expect the average node app has a lot more deep dependencies
13:31 < gmaxwell> sgeisler: at least a year to two ago that certantly wasn't always the case.  In any case, npm packages can be pinned too (and doesn't have moving language problems, even if they are much better in rust now), but in practice no one does.
13:32 < gmaxwell> gwillen: How deep do you need to be to get 0wned?  Go pick any rust package that uses http for anything, you'll see that it's pulled in a fair amount of seemingly inexplicable stuff.
13:32 < gmaxwell> sorry, correction, not 'any'-- there are a few rust people who do try to not pull in a dependency hell, but I think they're all in this room. :P anyone else...
13:33 < gmaxwell> I've seen andytoshi and bluematt walk away from using stuff because it has some mess of depenencies, but generally in rust you don't do that, thats why the whole packaging system is there.
13:34 < BlueMatt> yea, the rust ecosystem is generally not super great about it
13:35 < BlueMatt> certainly we're trying to do better at it in the rust-bitcoin ecosystem
13:35 < gwillen> the more dependencies you have, the larger the attack surface, so I expect rust software has less practical attack surface because I suspect it has fewer dependencies on average than node apps
13:35 < BlueMatt> and are somewhat anal about not pulling in deps
13:35 < gwillen> or have we forgotten the left pad incident
13:35 < sgeisler> In general DRY and using libraries are a good thing imo, there just had to be a way to peer review these :/
13:35 < BlueMatt> but, as gmaxwell points out, http in rust tends to pull in 30 deps
13:35 < BlueMatt> still much better than npm (leftpad....) but thats really more than bad enough for you to get poped
13:35 < gwillen> I think the left pad incident pretty clearly demonstrates how cavalier the node ecosystem is about pulling in numerous dependencies for stupid things
13:35 < BlueMatt> gwillen: heh, jinx
13:35 < gwillen> BlueMatt: I wanna get poped
13:36 < BlueMatt> .........
13:36 < gwillen> I just need to get some Cardinals to elect me
13:36 < gwillen> and then I get to wear a really cool hat
13:36 < BlueMatt> english is hard
13:36 < BlueMatt> or you can pull a luke and consider the papacy illegitimate, then you can (I suppose?) pick your own?
13:36 < gwillen> I would also be curious how many node apps vs rust apps are checking in their lockfile
13:37 < gmaxwell> not just 30 deps but also those 30 deps include a bunch of "what the @#$@ is this for" and "why would anyone ever make a library for this!?!" which is the npm ecosystem problem, if smaller.
13:37 < gwillen> to avoid dependency drift
13:37 < BlueMatt> anyway, we're trying to do it better in rust-bitcoin, but rust isn't *that* much better than node
13:37 < BlueMatt> I do hope it gets better with maturity
13:37 < BlueMatt> gwillen: its hard to pin in rust today :(
13:37 < gwillen> is checking in Cargo.lock not sufficient?
13:37 < BlueMatt> at least if you want to integrate at all with the rest of the ecosystem you kinda cant pin (ie upload to crates)
13:37 < BlueMatt> gwillen: no, it is not
13:37 < gwillen> you shouldn't check in a lockfile with a library, and you wouldn't upload an app to creates
13:38 < gwillen> crates*
13:38 < BlueMatt> errr, it is my understanding is that it is not
13:38 < BlueMatt> there is support for doing it at a build level
13:38 < gwillen> I believe checking in Cargo.lock with your application will save you (and doing so with a library is not correct)
13:38 < BlueMatt> which apparently liquid uses?
13:38 < BlueMatt> I havent tried it, though
13:38 < BlueMatt> cargo vendor or something
13:38 < sipa> gmaxwell: i assume at least that there is no rust package for testing whether a variable is an integer...
13:38 < gwillen> where's andytoshi, I'm sure he knows
13:38 < BlueMatt> yea, andrew would know
13:38 < BlueMatt> sipa: I....hope not
13:39 < sipa> https://www.npmjs.com/package/is-integer
13:39 < BlueMatt> indeed, the fact that the standard library provides....basic things means you dont end up with leftpad
13:39 < sipa> ... which depends on is-finite
13:40 < sipa> ... which depends on number-is-nan
13:40 < gwillen> see what I mean
13:40 < BlueMatt> sipa: holy fucking shit
13:40 < gwillen> not to be glib and insulting about node devs, but let me be glib and insulting about node devs: I think Rust has more of a culture of "writing software" and less of a culture of "slapping shit together with other shit into software"
13:40 < gwillen> and I suspect that shows in the dependency graph
13:41 < BlueMatt> it does
13:41 < BlueMatt> but the point is that even that is not sufficient
13:41 < BlueMatt> sure, instead of 10000 dependencies you have 100
13:41 < BlueMatt> but they're all bigger
13:41 < BlueMatt> and you still have to go audit them
13:41 < BlueMatt> which is super nontrivial
13:43 < sgeisler> Example: rocket (web framework) basic templating example dependencies https://gist.github.com/sgeisler/08a337bc614384682e56f0d48dc76f91
13:44 < sgeisler> (you can use cargo-tree to render such dependency graphs)
13:44 < BlueMatt> ouch
13:45 < gwillen> would be good to remove dupes, and remove things that are maintained by the rust core devs
13:45 < BlueMatt> "maintained by the rust core devs" isnt enough if you still have one guy's laptop that can upload a new build
13:45 < gwillen> some of these crates are like, part of the standard library
13:45 < BlueMatt> which is true for most of the "maintained by rust core devs" packages, afaict
13:46 < gwillen> if you're considering the standard library subject to this attack, _and_ you don't care how the attack surface scales beacuse any single package is enough...
13:46 < sgeisler> in comparision rust-bitcoin https://gist.github.com/18b542f301b744c654432057ab4bcfee or rust-lightning https://gist.github.com/c97620a89864fc1f27291d039ece9dba
13:46 < BlueMatt> no, but maintained pretty core to rust, yes, but still don't strictly have a good security story around getting the package and shit
13:46 < gwillen> you may as well go home at that point :-P
13:46 < BlueMatt> no? thats a very legit concern
13:46 < gwillen> I assume, given what I know about Rust and the Rust devs, that the security story is "you get it with cargo, cargo checks signatures or at least uses SS:"
13:47 < BlueMatt> sgeisler: and we're working on getting rid of rust-crypto, which is most of the dep tree there :P
13:47 < BlueMatt> gwillen: nope
13:47 < gwillen> if your "very legit concern" is about the stdlib being compromised, you might as well also assume the compiler may be compromised, and now we're looking at a TOTALLY different threat model from dependency blowup
13:47 < BlueMatt> I mean I can steal one guy's laptop and go edit the source then cargo publish
13:47 < BlueMatt> and boom you get backdoor'd
13:48 < instagibbs> "get poped" I read this as "popped" and was very confused
13:48 < gwillen> it was supposed to be popped
13:48 < BlueMatt> if at least there's a release process where you have several people looking thats way better
13:48 < gwillen> I do not want to get popped, but I do want to get poped (not really, it would be a lot of work)
13:48 < gwillen> BlueMatt: I agree, this is a real problem that should be fixed
13:48 < BlueMatt> yea, i mean thats my "maintained by guy X" objection
13:49 < BlueMatt> most of those packages dont have the kind of review that rustc/cargo/etc get
13:49 < BlueMatt> or release process
13:49 < BlueMatt> some of them are moving that direction, so thats just a maturity issue
13:49 < BlueMatt> but...
13:49 < gwillen> I think "steal a core dev's laptop" is a more difficult attack than "get the maintainer of is-multiple-of-seven to hand you the reins"
13:49 < BlueMatt> indeed, but we're not trying to just "do better than node", we want to actually "do well" :P
13:49 < gwillen> so I still think dependency bloat is a separate and important vector from adequate control of individual dependencies
13:49 < gwillen> and the fixes are basically orthogonal
13:50 < BlueMatt> the solution to both is just to not have dependencies :P
13:50 < gwillen> :-P
13:50 < gwillen> you are welcome to go write code on your desert island, BlueMatt
13:50  * sipa starts the NIH song
13:51 < BlueMatt> gwillen: I mean look at rust-bitcoin, turns out its minimal effort to go put together sha256 :P
13:51 < BlueMatt> though obviously no one is proposing writing a rust-bitcoin-http-client-nih
13:51 < gwillen> actually I think that's basically the plan
13:51 < gwillen> given that the http client is the main source of useless dependencies
13:51 < gwillen> and we are not using 99.97% of its features
13:52 < gwillen> and what we really need is just a way to spit the minimal set of HTTP headers, and hear the minimal set back
13:52 < BlueMatt> welllllll, ok, nvm
13:52 < BlueMatt> yea, but not a full http client
13:52 < gwillen> but in most cases it makes sense to depend on a library that does something interesting or complex that you actually _need_
13:52 < gwillen> right, yeah, just a stub/wrapper
13:53 < BlueMatt> anyway, point being the rust-bitcoin desert island turns out to not have been much work, though admittedly 90% of the dependency hell most people appear to fall into is gui shit, and we dont have gui shit
13:54 < sipa> there are GUIs in rust?
13:54 < sipa> (tell me to shut up where appropriate, i know nothing)
13:54 < gwillen> one thing that would save a lot of unnecessary dependency pain would be a good system for feature-flagging off dependencies that are necessary for features you don't want
13:54 < gwillen> that would probably save our usage of whatever the http crate is
13:54 < gwillen> since the dependencies are all for things we have no use for
13:55 < BlueMatt> sipa: there are like gtk wrappers, but I dont think anyone uses them
13:55 < BlueMatt> sipa: people (eg the firefox people) do use some opengl wrapper things, though
13:55 < BlueMatt> gwillen: yea, I've beaten andytoshi into submission on that one on rust-bitcoin and I think we're both really happy with it
13:55 < gwillen> *nods*
13:55 < BlueMatt> dont want serde? dont set the feature flag
13:55 < gwillen> it would be good if that were extremely easy to do, I dunno how it is now
13:56 < BlueMatt> its super duper easy
13:56 < BlueMatt> at a language level
13:56 < gwillen> also, one thing that the latest npm incident suggests to me: it would be good if you could automatically audit your dependencies for new _contributors_
13:56 < BlueMatt> but it does have a maintinence burden since you have to test a bunch of different configs
13:56 < gwillen> new code from old contributors could have security holes but it's less likely to be outright backdoored
13:56 < gwillen> but code from new contributors has a high index of suspicion
13:57 < gwillen> (obviously in the absence of signed commits, this only takes you so far, but if people start forging contributor names, that at least produces a big red flag as well)
13:57 < BlueMatt> yea, dunno if there's an api for "who can upload this package to npm"
13:58 < BlueMatt> sadly I dont think crates has nice access control yet
13:58 < BlueMatt> oh, actually, no they do have like shared accounts or whatever
13:58 < BlueMatt> wonder if thats api'd
13:58 < gwillen> well, I said 'new contributors' to also cover the case of commits from new users into an existing package
13:58 < gwillen> since those are also good to look at
13:58 < gwillen> a new uploader is even more suspiciosu
14:00 <@andytoshi> hi oops, i was not on IRC all day somehow. reading scrollback
14:06 <@andytoshi> a couple comments -- 1. If you commit Cargo.lock in a library it does basically nothing; cargo will actually ignore it when your users compiler their projects. (It will be used when you're building the library as a top-level project, e.g. in CI, but what good is that)
14:06 <@andytoshi> this is frustrating but the alternative would mean multiple versions of libraries in play, which in a strongly-typed language causes a lot of grief when the different versions' types are incompatible
14:09 <@andytoshi> 2. Rust does not encourage frivilous crates, though it does encourage splitting crates up (and provides absolutely no tooling to distinguish when many crates come from the same people or even the same git repo) which can result in large-looking dep trees. But there is also an anti-dependency subculture within the rust ecosystem, and most "serious" crates -- e.g. rust-nursery crates, byteorder,
14:09 <@andytoshi> etc., have few/no dependencies and take dependency creep seriously
14:10 <@andytoshi> unfortunately there is a large subset of the rust community who appear to have literally come from javascript, so everything related to web development is total chaos and bullshit, and i'm sure we'll see npm-style attacks sooner than later, and hopefully that will trigger the cargo developers to improve tooling
14:10 <@andytoshi> but it's very easy to avoid this entire subset of the community, with the major exception that everyone seems to think tokio is a reasonable replacement for select()
14:11 < BlueMatt> andytoshi: to be fair, tokio itself is actually well done and slowly moving towards first-class support (weel *very* slowly)
14:11 <@andytoshi> relatedly i sent this rant to oleg today about rust https://0bin.net/paste/A1pEatgkwUd9tk7f#Lcfc4dP5lhRpadEYK8f97xbf-6xzhq8hfflD7vGySh2
14:12 <@andytoshi> BlueMatt: oh, nice, but it still has a very big dep tree doesn't it?
14:12 <@andytoshi> this is fine i guess if eventually the tree rarely changes
14:12 < BlueMatt> no?
14:12 <@andytoshi> oh?
14:13 < BlueMatt> tokio depends on a million and one tokio crates
14:13 <@andytoshi> ah, yep, i see
14:13 <@andytoshi> i really wish cargo-tree had a way to treat that whole blob as one conceptual dependency
14:14 < BlueMatt> well luckily they're all prefixed by tokio
14:15 < BlueMatt> tokio depends on like 7 things, which isnt good, but its not that huge
14:15 < BlueMatt> mio, log, iovec, bytes, futures (which is kinda std-ish, soon), crossbeam, rand, and num_cpus
14:16 < BlueMatt> and slab
14:16 <@andytoshi> ok, nice, i've actually audited half of those, they're all pretty solid
14:17 < BlueMatt> serde is much worse
14:17 <@andytoshi> yeah, if serde wasn't as unsafe-averse as it is there's no chance i'd be using it, it's *huge* and super hard to read
14:18 < BlueMatt> oh, and apparently parking_lot now? There was some debate about moving the std mutex to the parking lot one, though
14:18 < BlueMatt> cause apparently the std mutex has undefined behavior they cant get rid of :(
14:18 <@andytoshi> yeah, i saw some of those discussions
14:18 <@andytoshi> but parking_lot is apparently different in some user-visible way so it'd be a breaking change
14:19 < BlueMatt> ah, damn
14:19 < BlueMatt> well hopefully they take part of it and then tokio can just drop its dep
14:19 <@andytoshi> the impression i get is that they will
14:19 <@andytoshi> eventually. it's a hard problem
14:20 < BlueMatt> yea, that was my impression
14:32 < stevenroose> I think a pretty good solution would be if the Cargo people make it easier to pin to git commits
14:33 < stevenroose> A lot of deps are quite final. And if they are final, and you are really not interested in any update, it should be the default way to commit to a hash, even for libraries.
14:34 < stevenroose> But right now if you want to circumvent crates.io by using git, your packages are incompatible. It should be possible to indicate to Cargo that the are compatible.y
15:52 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection]
15:56 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #rust-bitcoin
15:59 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection]
16:00 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #rust-bitcoin
16:35 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection]
16:38 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #rust-bitcoin
16:40 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection]
18:48 -!- _cryptodesktop_i [~John@91.245.77.21] has joined #rust-bitcoin
19:17 < ariard> BlueMatt: #198 is reviewable, was working on the suggested test
19:18 < ariard> but I'm stucking at some point on the lock in broadcast_transaction when I try to connect enough blocks to timeout the HTLC
19:18 < ariard> *one of the HTLC output
19:22 -!- _cryptodesktop_i [~John@91.245.77.21] has quit [Ping timeout: 268 seconds]
19:27 < BlueMatt> andytoshi: oh, btw, https://github.com/rust-bitcoin/rust-lightning/commit/3a066ccbf27ea249dcd7403530cda33318ebb3e7 cut about 10 minutes off our travis runs which do the fuzz steps
19:28 < BlueMatt> ariard: hmm, i'll take a look tomorrow
19:28 < BlueMatt> ariard: what do you mean stuck in the lock?
19:29 < BlueMatt> you mean the txn_broadcasted lock in the TestBroadcaster?
19:29 < BlueMatt> that isnt held across the function boundary so unless your test is holding the lock you shouldn't get stuck on it
19:29 < BlueMatt> make sure you arent doing something like let txn_broadcasted = ....txn_broadcasted.lock().unwrap();
19:29 < BlueMatt> outside of a tight {}
21:44 -!- wumpus [~ircclient@pdpc/supporter/professional/wumpus] has quit [Read error: Connection reset by peer]
21:44 -!- wumpus2 [~ircclient@pdpc/supporter/professional/wumpus] has joined #rust-bitcoin
23:37 -!- grubles [~grubles@unaffiliated/grubles] has quit [Quit: Leaving]
--- Log closed Tue Nov 27 00:00:35 2018