--- Day changed Tue Nov 19 2019 00:02 -!- EvaristeGalois [~quassel@unaffiliated/evaristegalois] has joined #lnd 00:04 -!- pioklo [~Pioklo@118-40.echostar.pl] has joined #lnd 00:44 -!- belcher [~belcher@unaffiliated/belcher] has joined #lnd 01:27 -!- jonatack [~jon@2a01:e35:8aba:8220:6627:dad:d967:649d] has quit [Ping timeout: 276 seconds] 02:02 -!- jonatack [~jon@abayonne-654-1-191-177.w92-134.abo.wanadoo.fr] has joined #lnd 02:11 -!- jonatack [~jon@abayonne-654-1-191-177.w92-134.abo.wanadoo.fr] has quit [Ping timeout: 240 seconds] 02:13 -!- jonatack [~jon@109.232.227.138] has joined #lnd 02:16 -!- gethh [uid264798@gateway/web/irccloud.com/x-ipkfxywxxabxawkd] has quit [Quit: Connection closed for inactivity] 02:26 -!- libertyprime [~libertypr@124.197.60.232] has joined #lnd 02:28 -!- pioklo [~Pioklo@118-40.echostar.pl] has quit [Ping timeout: 240 seconds] 03:04 -!- pioklo [~Pioklo@backup.linuxpl.com] has joined #lnd 03:16 -!- ironbutt [~LiberLive@144.49.211.130.bc.googleusercontent.com] has quit [Remote host closed the connection] 03:35 -!- jonatack [~jon@109.232.227.138] has quit [Ping timeout: 250 seconds] 03:42 -!- whitez00 [whitez00@gateway/vpn/protonvpn/whitez00] has joined #lnd 03:52 -!- pioklo [~Pioklo@backup.linuxpl.com] has quit [Ping timeout: 245 seconds] 04:19 -!- pioklo [~Pioklo@ip-91.246.70.194.skyware.pl] has joined #lnd 04:33 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 04:33 < lnd-bot> [lnd] joostjager opened pull request #3742: invoices: expose custom tlv records from the payload (master...expose-custom-tlv) https://github.com/lightningnetwork/lnd/pull/3742 04:34 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 04:45 -!- thomasanderson [~thomasand@108-91-143-73.lightspeed.rlghnc.sbcglobal.net] has joined #lnd 04:54 -!- rh0nj [~rh0nj@88.99.167.175] has quit [Remote host closed the connection] 04:54 -!- _jerry [~jerry@47.55.112.140] has joined #lnd 04:55 -!- rh0nj [~rh0nj@88.99.167.175] has joined #lnd 05:26 -!- slivera__ [slivera@gateway/vpn/privateinternetaccess/slivera] has quit [Quit: Leaving] 05:49 -!- thomasanderson [~thomasand@108-91-143-73.lightspeed.rlghnc.sbcglobal.net] has quit [Ping timeout: 265 seconds] 05:56 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 05:56 < lnd-bot> [lnd] halseth merged pull request #2520: specify a tag, branch or commit during docker prod build (master...choose-git-state-during-docker-prod-build) https://github.com/lightningnetwork/lnd/pull/2520 05:56 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 05:56 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 05:56 < lnd-bot> [lnd] halseth pushed 2 commits to master: https://github.com/lightningnetwork/lnd/compare/a00a36001800...79051ac63f1a 05:56 < lnd-bot> lnd/master 3173680 yancy ribbens: docker: specify a tag, branch or commit using docker build-arg 05:56 < lnd-bot> lnd/master 79051ac Johan T. Halseth: Merge pull request #2520 from yancyribbens/choose-git-state-during-docker-... 05:56 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 06:00 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has quit [Remote host closed the connection] 06:02 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has joined #lnd 06:28 -!- gethh [uid264798@gateway/web/irccloud.com/x-sdklpkedmpczudpk] has joined #lnd 07:01 -!- Pioklo_ [~Pioklo@ip-91.246.70.194.skyware.pl] has joined #lnd 07:01 -!- pioklo [~Pioklo@ip-91.246.70.194.skyware.pl] has quit [Read error: Connection reset by peer] 07:27 -!- mdunnio [~mdunnio@38.126.31.226] has joined #lnd 07:46 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Remote host closed the connection] 07:48 -!- mdunnio [~mdunnio@38.126.31.226] has joined #lnd 08:07 -!- whitez00 [whitez00@gateway/vpn/protonvpn/whitez00] has quit [Quit: Konversation terminated!] 08:33 -!- vincenzopalazzo [~vincenzop@2.226.158.237] has joined #lnd 09:08 -!- Talkless [~Talkless@hst-227-49.splius.lt] has joined #lnd 09:32 -!- openoms [~quassel@185.174.156.219] has joined #lnd 09:34 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Remote host closed the connection] 09:46 -!- mdunnio [~mdunnio@38.126.31.226] has joined #lnd 09:52 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Ping timeout: 276 seconds] 10:00 -!- mdunnio [~mdunnio@38.126.31.226] has joined #lnd 10:28 -!- vincenzopalazzo [~vincenzop@2.226.158.237] has quit [Quit: Leaving] 11:06 -!- willcl_ark [~quassel@cpc123762-trow7-2-0-cust7.18-1.cable.virginm.net] has quit [Quit: Quit] 11:19 -!- willcl_ark [~quassel@cpc123762-trow7-2-0-cust7.18-1.cable.virginm.net] has joined #lnd 11:57 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Remote host closed the connection] 11:59 -!- mdunnio [~mdunnio@38.126.31.226] has joined #lnd 12:07 -!- Talkless [~Talkless@hst-227-49.splius.lt] has quit [Quit: Konversation terminated!] 12:40 -!- jtimon [~quassel@22.133.134.37.dynamic.jazztel.es] has joined #lnd 12:50 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Remote host closed the connection] 12:52 -!- mdunnio [~mdunnio@38.126.31.226] has joined #lnd 13:26 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 13:26 < lnd-bot> [lnd] guggero closed pull request #3690: chanrestore: assume height hint for unconfirmed channels in SCBs (master...zero-height-hint) https://github.com/lightningnetwork/lnd/pull/3690 13:26 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 13:41 -!- spoke0 [~spoke0@92.116.116.27] has joined #lnd 14:05 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has quit [Remote host closed the connection] 14:06 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has joined #lnd 14:24 -!- slivera [slivera@gateway/vpn/privateinternetaccess/slivera] has joined #lnd 14:42 -!- spoke0 [~spoke0@92.116.116.27] has quit [Read error: Connection reset by peer] 14:42 -!- spoke0_ [~spoke0@92.116.116.27] has joined #lnd 15:13 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Remote host closed the connection] 15:24 -!- EvaristeGalois [~quassel@unaffiliated/evaristegalois] has quit [Remote host closed the connection] 15:42 -!- Pioklo_ [~Pioklo@ip-91.246.70.194.skyware.pl] has quit [Ping timeout: 240 seconds] 15:44 -!- spoke0_ [~spoke0@92.116.116.27] has quit [Quit: Leaving...] 15:48 -!- spoke0 [~spoke0@92.116.116.27] has joined #lnd 16:02 -!- slivera [slivera@gateway/vpn/privateinternetaccess/slivera] has quit [Remote host closed the connection] 16:21 -!- spoke0_ [~spoke0@144.76.166.209] has joined #lnd 16:24 -!- spoke0 [~spoke0@92.116.116.27] has quit [Ping timeout: 240 seconds] 16:44 -!- libertyprime [~libertypr@124.197.60.232] has quit [Ping timeout: 246 seconds] 16:52 -!- spoke0 [~spoke0@92.116.116.27] has joined #lnd 16:52 <@roasbeef> GB's def sounds super off eugene 16:52 <@roasbeef> i run it np on my phone 16:52 <@roasbeef> you're measure resident memory usage? 16:52 <@roasbeef> iirc we also sort first in a few areas 16:52 -!- spoke0 [~spoke0@92.116.116.27] has quit [Read error: Connection reset by peer] 16:53 -!- spoke0 [~spoke0@92.116.116.27] has joined #lnd 16:53 -!- lndbot1 [~lndbot@138.197.213.35] has quit [Remote host closed the connection] 16:53 <@roasbeef> or you mean raw I/O? 16:53 -!- lndbot [~lndbot@138.197.213.35] has joined #lnd 16:53 < lndbot> is the bridge doing any better now? 16:53 < lndbot> you should get a proper IRC client @eugene;) 16:53 < lndbot> ok back 16:53 < lndbot> my problem with IRC is that it never keeps logs 16:53 <@roasbeef> you keep your own logs ;) 16:53 < lndbot> ah 16:53 <@roasbeef> and scrollback 16:54 <@roasbeef> i have a VM that only does IRC lol 16:54 <@roasbeef> then mosh -> tmux -> irssi 16:54 < lndbot> oh i see, i might get that setup then 16:54 <@roasbeef> and I use juice ssh on my phone, it supports mosh 16:54 < lndbot> do u have a rooted droid 16:54 <@roasbeef> nope 16:54 <@roasbeef> stock 16:55 < lndbot> Ah gotcha 16:56 < lndbot> This only happens on startup for me the very first time. The linked issue in bbolt describes it sort of accurately. Basically new buckets with a lot of `Put` calls with seemingly "random" writes (whatever that means?) cause pages to split and huge copying of what bbolt calls "inodes" 16:56 -!- spoke0_ [~spoke0@144.76.166.209] has quit [Ping timeout: 276 seconds] 16:56 <@roasbeef> like you actually see GBs of resident memory being allocated? have you profiled this? 16:57 < lndbot> It's with pprof 16:57 <@roasbeef> i don't think that's the case, otherwise android would kill it pretty quickly 16:57 <@roasbeef> in terms of actual resident memory that's paged in 16:57 < lndbot> The log I posted was with pprof 16:57 < lndbot> I didnt bother to check the output of `ps` or `top` 16:58 < bitconner> eugene: which log? can you link again? 16:59 < lndbot> here i'll put it in a gist 16:59 -!- fox2p_ [~fox2p@cpe-66-108-40-154.nyc.res.rr.com] has quit [Ping timeout: 240 seconds] 17:01 < lndbot> https://gist.github.com/Crypt-iQ/634faf8857f31453c0c37f6309f2474a 17:02 -!- fox2p [~fox2p@cpe-66-108-32-173.nyc.res.rr.com] has joined #lnd 17:03 < lndbot> output from ps aux | grep lnd: nsa 96218 17.4 11.6 7134868 1953492 s006 U+ 7:58PM 1:11.83 lnd --rpclisten=localhost:10002 17:03 < lndbot> RSS is 1953492 KiB 17:08 < lndbot> This is what the creator of bbolt had to say: https://github.com/boltdb/bolt/issues/456#issuecomment-155986389 17:09 < bitconner> so atm there's about 75 MB of block and header filters that need to be written. write amplification due to b+ would explain some of that, but 216x seems rather high. have we confrimed that we're not overwriting existing data/ 17:09 < bitconner> ? 17:10 < lndbot> That's one of the first things I checked 17:10 < lndbot> Not overwriting filters from what I can tell 17:11 < bitconner> how big is the resulting db/ 17:12 < bitconner> also freelist sync or no freelist sync? 17:12 < lndbot> I think it's always 128mb or so? 17:12 < lndbot> I just deleted it 17:12 < lndbot> freelist sync, but most of the calls aren't from the freelist calls 17:15 < lndbot> Here one sec 17:15 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 17:15 < lnd-bot> [lnd] Roasbeef merged pull request #3740: lnd: increase max message receive size for REST proxy (master...rest-client-receive-size) https://github.com/lightningnetwork/lnd/pull/3740 17:15 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 17:15 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 17:15 < lnd-bot> [lnd] Roasbeef pushed 2 commits to master: https://github.com/lightningnetwork/lnd/compare/79051ac63f1a...84c73f302500 17:15 < lnd-bot> lnd/master 4cd0646 Oliver Gugger: lnd: increase max message receive size for REST proxy 17:15 < lnd-bot> lnd/master 84c73f3 Olaoluwa Osuntokun: Merge pull request #3740 from guggero/rest-client-receive-size 17:15 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 17:15 < bitconner> final size sounds about right. hmm i wonder what the difference would be with no freelist sync, hard to tell from the prior pprof. we could just be looking at write amplification, in case we'd want to reduce the number of overall keys used 17:16 < lndbot> https://gist.github.com/Crypt-iQ/8150bc21ebe98cd65a9e986ccff18bd2 17:16 < lndbot> I think it's write amplification 17:17 < lndbot> I mean there's about 2gigs of "allocate" calls which could probably be reduced with no freelist sync 17:17 < lndbot> But that gist I just posted shows what happens with a Put call 17:17 -!- vindard [~vindard@190.83.165.233] has quit [Ping timeout: 245 seconds] 17:20 < bitconner> the bigger issue looks to be that bbolt only extends the inodes list by one inode at a time, instead of using double and copy 17:21 < bitconner> if thats the case it would result in a quadratic amount of memory, which would make sense in this case... 17:29 -!- spoke0 [~spoke0@92.116.116.27] has quit [Quit: Leaving...] 17:30 -!- spoke0 [~spoke0@92.116.116.27] has joined #lnd 17:50 < lndbot> I wonder why the append call is taking so much memory? 17:53 -!- vindard [~vindard@190.83.165.233] has joined #lnd 17:53 < bitconner> my guess is probably bc the list is full, so it allocates a completely new one with size n+1, then copies everything over 17:54 < bitconner> j 17:54 < lndbot> Ahh right 17:58 < bitconner> did hacky fix and benchmark to compare https://paste.ubuntu.com/p/V2XWfzGvc3/ 18:01 < bitconner> test inserts 600k entries just like the header entry per iteration, though into the root bucket instead of one level deep 18:06 < lndbot> nice 18:06 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 18:06 < lnd-bot> [lnd] Roasbeef merged pull request #1703: Replace maintainer by label in Dockerfiles (master...master) https://github.com/lightningnetwork/lnd/pull/1703 18:06 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 18:06 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 18:06 < lnd-bot> [lnd] Roasbeef pushed 1 commit to master: https://github.com/lightningnetwork/lnd/compare/84c73f302500...b03d8edcd9ad 18:06 < lnd-bot> lnd/master b03d8ed Philipp Gillé: docker: replace maintainer by label in Dockerfiles 18:06 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 18:07 -!- libertyprime [~libertypr@client-173-018.guest.otago.ac.nz] has joined #lnd 18:07 < lndbot> Why are there only 2 allocs in the new case? 18:08 < lndbot> I left a comment here where `childAt` calls `read` eventually 18:08 < lndbot> https://gist.github.com/Crypt-iQ/8150bc21ebe98cd65a9e986ccff18bd2#gistcomment-3087894 18:09 < lndbot> Wait but your 600k entries only used 22MB??? 18:10 < bitconner> yes hash + height is 36 bytes in total 18:11 < lndbot> Is that stat the number of allocated bytes? 18:11 < lndbot> What if you run the test with -memprofile? 18:16 < bitconner> about the same https://paste.ubuntu.com/p/tB8j9DDH8T/ 18:16 < bitconner> note that those numbers aren't per iteration, they're totals 18:17 < bitconner> there could be other things that aren't captured by the benchmark, separating calls into separate txns, different level in the tree, etc 18:18 < bitconner> seems promising tho, can def see how that would be a hot spot 18:19 < lndbot> Oh I see... hmm so that's for the total number of calls in the benchmark 18:19 < lndbot> Are you making a new bucket each time? 18:21 < bitconner> no that's only adding to the root bucket, neutrino actually inserts them one level lower 18:21 < lndbot> Hm yeah 18:22 < bitconner> my guess is that inserting lower would make the disparity greater, but haven't tried 18:22 < lndbot> hmm 18:23 < lndbot> Do you know what N is? 18:23 < bitconner> N in what? 18:23 < lndbot> the benchmar 18:23 < lndbot> benchmark 18:24 < bitconner> the final pass did 9 iterations, that's the first number in the benchmark output 18:25 < bitconner> oh nvm, i used benchcmp so that doesn't show up 18:26 < lndbot> Hm I wonder why you don't get my crazy numbers 18:26 < lndbot> I don't think all the insertions are on the same node 18:27 < lndbot> There's a call to c.node().put(...) in Bucket.Put() and so it seems to me that put(...) would be called on different nodes 18:27 < bitconner> yeah that's what i mean by this benchmark probably doesn't replicate the exact mechanics of how its used in neutrino. when the tx commits it probably does some splitting and other things, which would be different from the contrived example here 18:28 < lndbot> ok cool well still a great example 18:28 < lndbot> your fix did allocate less bytes :P 18:33 < lndbot> I do think neutrino could benefit from no freelist sync 18:33 < lndbot> Might be worth a PR 18:35 < bitconner> at one point we added it, and then i think we disabled it bc startup was pretty slow on some devices 18:36 < lndbot> Ohhh that makes sense 18:36 < lndbot> Since neutrino is exclusively mobile 18:57 -!- libertyprime [~libertypr@client-173-018.guest.otago.ac.nz] has quit [Ping timeout: 240 seconds] 18:59 -!- libertyprime [~libertypr@client-172-206.guest.otago.ac.nz] has joined #lnd 19:15 -!- spoke0_ [~spoke0@92.116.125.101] has joined #lnd 19:18 -!- spoke0 [~spoke0@92.116.116.27] has quit [Ping timeout: 250 seconds] 19:26 -!- reallll [~belcher@unaffiliated/belcher] has joined #lnd 19:29 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 240 seconds] 19:49 -!- libertyprime [~libertypr@client-172-206.guest.otago.ac.nz] has quit [Ping timeout: 246 seconds] 19:50 -!- lnd-bot [~lnd-bot@165.227.7.29] has joined #lnd 19:50 < lnd-bot> [lnd] cfromknecht opened pull request #3743: lnwire: assert sorted short channel ids (master...in-order-sids) https://github.com/lightningnetwork/lnd/pull/3743 19:50 -!- lnd-bot [~lnd-bot@165.227.7.29] has left #lnd [] 19:51 -!- libertyprime [~libertypr@client-172-206.guest.otago.ac.nz] has joined #lnd 21:50 -!- _jerry [~jerry@47.55.112.140] has quit [Ping timeout: 240 seconds] 21:50 -!- jtimon [~quassel@22.133.134.37.dynamic.jazztel.es] has quit [Ping timeout: 245 seconds] 22:55 -!- spoke0_ [~spoke0@92.116.125.101] has quit [Read error: Connection reset by peer] 22:55 -!- spoke0 [~spoke0@92.116.125.101] has joined #lnd 22:56 -!- spoke0 [~spoke0@92.116.125.101] has quit [Read error: Connection reset by peer] 22:56 -!- spoke0_ [~spoke0@92.116.125.101] has joined #lnd 23:37 -!- openoms [~quassel@185.174.156.219] has quit [Ping timeout: 265 seconds]