--- Log opened Thu Dec 12 00:00:34 2019
00:53 -!- designwish [~designwis@51.ip-51-68-136.eu] has joined #c-lightning
00:59 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
01:03 -!- MasterdonX [~masterdon@104.200.131.6] has quit [Ping timeout: 268 seconds]
01:05 -!- belcher [~belcher@unaffiliated/belcher] has quit [Ping timeout: 265 seconds]
01:07 -!- MasterdonX [~masterdon@45.9.250.100] has joined #c-lightning
01:10 -!- kanzure [~kanzure@unaffiliated/kanzure] has quit [Ping timeout: 246 seconds]
01:12 -!- kanzure [~kanzure@unaffiliated/kanzure] has joined #c-lightning
01:20 -!- belcher [~belcher@unaffiliated/belcher] has joined #c-lightning
01:31 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 268 seconds]
01:35 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
01:37 -!- vasild_ [~vd@gateway/tor-sasl/vasild] has joined #c-lightning
01:40 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 260 seconds]
01:58 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 276 seconds]
02:02 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
02:02 -!- rusty [~rusty@pdpc/supporter/bronze/rusty] has quit [Quit: Leaving.]
02:06 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 245 seconds]
02:09 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
02:18 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has quit [Ping timeout: 245 seconds]
02:37 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 268 seconds]
02:40 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
02:41 < vasild_> ps -p 20981 -o vsz,rss,command
02:41 < vasild_>         VSZ    RSS COMMAND
02:41 < vasild_> 21475092800 695820 lightningd --daemon
02:42 < vasild_> looks like lightningd has allocated 21 terabytes and uses 695MB of them, is this expected?
02:43 -!- vasild_ is now known as vasild
02:44 -!- rusty [~rusty@pdpc/supporter/bronze/rusty] has joined #c-lightning
02:47 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-nfufoseoysnzgsxq] has left #c-lightning []
02:47 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-nfufoseoysnzgsxq] has joined #c-lightning
02:55 <@cdecker> vasild: no, that's not expected. Is that a really busy node?
02:56 < m-schmoock> maybe instead of busy its a greedy node :D
02:59 < vasild> It has just one channel opened and no activity on it. I just started playing with it and couldn't believe my eyes when I saw 20T in the "size" column in top(1).
03:01 < vasild> I noticed also that lightningd was staying at 50% CPU for a long time, maybe more than an hour, this is what prompted me to look at top(1). Now its CPU usage has dropped to a normal idling state, e.g. ~0.1%
03:01 < m-schmoock> hm. mine is just    196832 178144 /home/btc/bin/lightningd --daemon
03:01 < m-schmoock> what version and network are you running?
03:01 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 265 seconds]
03:02 < vasild> ok, I just wanted to check if it is doing some black magic and the insane vsz is expected, but it is not :)
03:02 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
03:02 < m-schmoock> i have 20 open channels or so
03:03 < m-schmoock> you?
03:03 <@cdecker> Hm, if you have built the daemon with `DEVELOPER=1` you can do a `lightning-cli dev-memleak` and see what is hogging memory
03:03 < vasild> fcbd11f0 (latest from git from a few days ago), bit mainnet, 1 open channel, nothing is being sent or received on it.
03:05 < vasild> I tried to compile in developer mode, but it failed with some linking errors "undefined reference to main and other functions", I have not investigated yet
03:05 < m-schmoock> cdecker: it is reserving far more than using, a memleak would also 'use' the memory
03:06 <@cdecker> Interesting
03:06 < vasild> I guess the 20TB would be easy to nail down, if it keeps happening over restart, given I have some free time to stare at it.
03:06 <@cdecker> Still ~700MB of real usage is not expected
03:07 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 246 seconds]
03:10 < m-schmoock> true
03:10 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
03:12 < vasild> also, the log was full with those: https://paste.gg/p/anonymous/6d19608d6fa24c22bf2cf99dff13a0bb, many times per second. That looked abnormal too.
03:14 < m-schmoock> how much RAM does your node have and for how long did it run?
03:15 < vasild> 32GiB RAM, it ran less than a day - a few hours.
03:17 < m-schmoock> how many open channels?
03:17 < vasild> I just restarted the lightningd process
03:17 < vasild>         VSZ    RSS %CPU COMMAND
03:17 < vasild> 21475028448 296932 15.6 lightningd --daemon
03:17 < vasild> good
03:17 < vasild> 1 open channel
03:17 < m-schmoock> lol
03:17 < m-schmoock> greedy daemon
03:17 < m-schmoock> not normal
03:17 < vasild> so it is reproducable, excellent
03:18 < m-schmoock> try this as cdecker pointed out: `DEVELOPER=1` rebuild, restart and run `lightning-cli dev-memleak`
03:20 < vasild> yes, and will fix developer compilation
03:21 < m-schmoock> cdecker: when/how is the `our_amount_msat` value updated in `listpeers` or `listfunds` ? I figured I need to wait/sleep a bit after making a payment (via plugin) before checking those values actually changed
03:22 <@cdecker> They're updated once the HTLC sending the funds away is committed to iirc
03:23 < m-schmoock> So the pay plugin doenst wait for this, right?
03:26 < m-schmoock> (sendpay command)
03:26 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 240 seconds]
03:27 < m-schmoock> *waitsendpay
03:28 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
03:37 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 240 seconds]
03:39 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #c-lightning
03:42 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
03:54 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 268 seconds]
03:54 -!- jonatack [~jon@213.152.162.104] has joined #c-lightning
04:00 -!- queip [~queip@unaffiliated/rezurus] has joined #c-lightning
04:02 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has joined #c-lightning
04:03 -!- cryptoso- [~cryptosoa@gateway/tor-sasl/cryptosoap] has quit [Ping timeout: 260 seconds]
04:05 -!- queip [~queip@unaffiliated/rezurus] has quit [Ping timeout: 240 seconds]
04:45 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 240 seconds]
04:52 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #c-lightning
05:48 -!- jonatack [~jon@213.152.162.104] has quit [Ping timeout: 240 seconds]
05:53 -!- lowentropy [~lowentrop@gateway/tor-sasl/lowentropy] has quit [Ping timeout: 260 seconds]
05:54 -!- lowentropy [~lowentrop@gateway/tor-sasl/lowentropy] has joined #c-lightning
06:19 -!- belcher [~belcher@unaffiliated/belcher] has quit [Quit: Leaving]
06:57 -!- jonatack [~jon@37.166.226.254] has joined #c-lightning
07:00 -!- belcher [~belcher@unaffiliated/belcher] has joined #c-lightning
07:13 -!- mol [~molly@unaffiliated/molly] has joined #c-lightning
07:50 -!- mdunnio [~mdunnio@38.126.31.226] has joined #c-lightning
07:53 -!- jonatack [~jon@37.166.226.254] has quit [Read error: Connection reset by peer]
08:36 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has joined #c-lightning
09:04 -!- Gabotron [~satoshi@41.red-2-139-189.staticip.rima-tde.net] has joined #c-lightning
09:08 -!- Gabotron [~satoshi@41.red-2-139-189.staticip.rima-tde.net] has quit [Quit: leaving]
09:56 -!- lio17 [~lio17@80.ip-145-239-89.eu] has quit [Quit: lio17]
09:57 -!- lio17 [~lio17@80.ip-145-239-89.eu] has joined #c-lightning
09:58 < jb55> looks like gossipd uses the most memory on my node: 126M resident, only 133M virtual. all the other daemons use almost no memory.
10:01 -!- lxer [~lx@ip5f5bd664.dynamic.kabel-deutschland.de] has joined #c-lightning
10:10 < m-schmoock> that is as expected i guess
10:14 < m-schmoock> my gossip also takes 115M and the rest close to nothing.
10:31 -!- jtimon [~quassel@22.133.134.37.dynamic.jazztel.es] has joined #c-lightning
10:33 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Remote host closed the connection]
10:37 -!- mdunnio [~mdunnio@38.126.31.226] has joined #c-lightning
10:58 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-nfufoseoysnzgsxq] has left #c-lightning []
10:58 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-nfufoseoysnzgsxq] has joined #c-lightning
11:02 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Remote host closed the connection]
11:08 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has quit [Ping timeout: 276 seconds]
11:12 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has joined #c-lightning
11:13 -!- mol [~molly@unaffiliated/molly] has quit [Ping timeout: 265 seconds]
11:14 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 268 seconds]
11:33 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has quit [Ping timeout: 252 seconds]
11:34 -!- jonatack [~jon@2a01:e0a:53c:a200:bb54:3be5:c3d0:9ce5] has joined #c-lightning
12:00 -!- mdunnio [~mdunnio@38.126.31.226] has joined #c-lightning
12:36 < vasild> Recompiling without the address sanitizer made it use the sane 38MB VSZ, instead of 20TB. I had forgotten that I compiled with ASAN.
13:00 -!- mdunnio [~mdunnio@38.126.31.226] has quit [Remote host closed the connection]
13:10 < jb55> that would do it...
13:27 -!- rusty [~rusty@pdpc/supporter/bronze/rusty] has quit [Quit: Leaving.]
13:40 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 260 seconds]
13:42 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #c-lightning
14:44 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #c-lightning
15:09 -!- achow101_ [~achow101@unaffiliated/achow101] has joined #c-lightning
15:09 -!- MasterdonX [~masterdon@45.9.250.100] has quit [Quit: ZNC 1.7.4 - https://znc.in]
15:10 -!- achow101 [~achow101@unaffiliated/achow101] has quit [Read error: Connection reset by peer]
15:10 -!- MasterdonX [~masterdon@45.9.250.100] has joined #c-lightning
15:37 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 276 seconds]
15:49 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has joined #c-lightning
16:09 -!- lxer [~lx@ip5f5bd664.dynamic.kabel-deutschland.de] has quit [Ping timeout: 265 seconds]
16:17 -!- jtimon [~quassel@22.133.134.37.dynamic.jazztel.es] has quit [Remote host closed the connection]
17:01 -!- rusty [~rusty@pdpc/supporter/bronze/rusty] has joined #c-lightning
17:02 < rusty> niftynei: around?
17:40 -!- Chris_Stewart_5 [~chris@unaffiliated/chris-stewart-5/x-3612383] has quit [Ping timeout: 246 seconds]
18:04 -!- Kostenko [~Kostenko@2001:8a0:729a:3f00:c298:ebb2:64cf:151] has quit [Ping timeout: 250 seconds]
18:19 -!- Kostenko [~Kostenko@2001:8a0:7292:c400:4cc5:1221:6621:1e88] has joined #c-lightning
18:31 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
19:04 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-nfufoseoysnzgsxq] has left #c-lightning []
19:05 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-nfufoseoysnzgsxq] has joined #c-lightning
20:05 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
20:05 -!- DeanWeen [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
20:16 < rusty> niftynei: ouch, generated code parsing bug on arrays...
21:31 -!- Victor_sueca [~Victorsue@unaffiliated/victorsueca] has joined #c-lightning
21:34 -!- Victorsueca [~Victorsue@unaffiliated/victorsueca] has quit [Ping timeout: 250 seconds]
22:05 <@niftynei> just took a look rusty, nice catch. 
22:07 <@niftynei> i actually ran into the test_corruption problem with init messages today and decided to skip on fixing it so...
22:07 <@niftynei> glad to see it's been taken care of :)
22:36 -!- rusty [~rusty@pdpc/supporter/bronze/rusty] has quit [Quit: Leaving.]
23:53 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has quit [Remote host closed the connection]
23:53 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Remote host closed the connection]
23:53 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has joined #c-lightning
23:53 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #c-lightning
--- Log closed Fri Dec 13 00:00:35 2019