--- Log opened Fri May 24 00:00:22 2019 00:29 < m-schmoock> niftynei: You can run a single test if that is suitable to the situation. How long does a full run take for you? 00:31 < m-schmoock> Turn of VALGRIND, use more threads (i use PYTEST_PAR=24), get a better CPU :D 00:43 < m-schmoock> cdecker: the situation with the completely drained channels is even worse if the remote side is a clightning node, as it will unilaterally close the channel as soon as you try to fill it up again 00:44 < m-schmoock> channel 4b01f6b9ec32073ab03827f07d91eece1253c90e858e0920d92f735fcd3010be: update_fee 253 unaffordable 00:45 < m-schmoock> tried this two times on testnet 00:48 < m-schmoock> the LND channels can recover if unlocked by a tiny payment that for some reason cause a lower (about half) HTLC commitment fee 01:00 < t0mix> roasbeef, if CL send invalid commit point, closure would be unilateral on CL side? 01:06 < m-schmoock> t0mix: he is surely still asleep right now 01:06 < m-schmoock> :D 01:06 < t0mix> well I just woke up =D 01:07 < t0mix> m-schmoock, your node is offline? 01:08 < m-schmoock> nopew 01:08 < m-schmoock> still making about 10 forwards a day 01:08 < m-schmoock> ├┼┤ 575708x2039x0 [__] t0mix 01:09 < m-schmoock> BillyGarrison and Zoltan are offline 01:09 < t0mix> ├──┼──┤ 575708x2039x0 [_O] schmoock.net 01:09 < m-schmoock> strange .... 01:09 < m-schmoock> lol 01:10 < m-schmoock> wait, let me try to use our channel 01:10 < m-schmoock> now it shows you offline 01:10 < m-schmoock> summary shows offline, but listpeers shows ""CHANNELD_NORMAL:Reconnected, and reestablished." 01:11 < m-schmoock> can yu confirm this? 01:11 < t0mix> confirmed, but "connected" : false 01:11 < m-schmoock> so reconnected != connected ?? 01:11 < t0mix> when I do connect, it works.. but conneted remain false anyway 01:12 < m-schmoock> hum 01:12 < m-schmoock> what a mess 01:12 < m-schmoock> so many bugs 01:12 < t0mix> I did upgrade yesterday, to master. since then I can see all channels =D 01:12 < m-schmoock> gz ^^ 01:13 < t0mix> oh, I can see your node online now! 01:15 < m-schmoock> for me your are offline again 01:16 < t0mix> yeah, same for me, now. wtf 01:17 < m-schmoock> I cant use our channel either. try to rebalance through it without success 01:17 < m-schmoock> uuuuh I get stacktraces °_° 01:17 < m-schmoock> gossip_store: can't read len 3330865322 offset 1823449 (version v0.7.0-332-gd054fef-modded) 01:17 < m-schmoock> thats unfortunate 01:20 -!- bitonic-cjp [~bitonic-c@92-111-70-106.static.v4.ziggozakelijk.nl] has joined #c-lightning 01:20 < t0mix> version v0.7.0-332-gd054fef-modded is including fix https://github.com/ElementsProject/lightning/pull/2505 ? 01:22 < m-schmoock> yes its master with a fix that is on master. did the mod before the pr was merges. so its basically unmodded 01:22 < m-schmoock> its basically exactly this code d9db9dc1 01:23 < m-schmoock> i will update and retry.... 01:26 < m-schmoock> online again 01:26 < t0mix> yes! 01:29 < m-schmoock> its having a hard time use your channel... but that is normal (for your node). Ill try with 10 minutes retry 01:29 < m-schmoock> eventually it worked everytime. just needs more tries 01:29 < t0mix> I wonder why 01:30 < m-schmoock> this time: 214ec84c84827dd4911de56d2ecb77d367c6f24c658b8acfe4826b01968e45594 chan #3628: failed htlc 78 code 0x400f (WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS 01:31 < m-schmoock> i dont really know, thats special to the channel between our nodes. 01:32 < m-schmoock> lucifer and codeoholic have latency issues as they are in south america and india. so no big deal. 01:32 < m-schmoock> and your node is needing more massage, more tries when using 01:32 < m-schmoock> all the remaining channels behave normal for me 01:32 < t0mix> I lost channels with lucifer and codeholic yesterday during restarts :( 01:32 < m-schmoock> t0mix: now it worked 01:42 -!- darosior [6dbe8dc1@gateway/web/freenode/ip.109.190.141.193] has joined #c-lightning 01:59 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has quit [Ping timeout: 272 seconds] 02:13 -!- sstone [~sstone@irc.acinq.fr] has joined #c-lightning 02:14 -!- sstone [~sstone@irc.acinq.fr] has quit [Remote host closed the connection] 02:18 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #c-lightning 02:26 -!- ulrichard [~richi@dzcpe6300borminfo01-e0.static-hfc.datazug.ch] has joined #c-lightning 03:07 <@cdecker> niftynei: you may have some luck with letting pytest use /dev/shm as a test directory (that's a ram-disk) 03:07 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...] 03:08 <@cdecker> I had a patch somewhere that would make the test_base_dir configurable but I can't find it atm 03:08 <@cdecker> Just change it manually in tests/fixtures.py for now :-) 03:11 <@cdecker> m-schmoock: WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS that means that there is no matching invoice for the payment, could have expired 03:20 < m-schmoock> its t0mix node that is behaving unsteady. all the others are fine. not sure about that RasPi support :D 03:21 <@cdecker> Could also be related to sync height (CLTV will mismatch if he is still syncing) 03:21 <@cdecker> Try to compare `bitcoin-cli getnetworkinfo` with the height displayd by `lightning-cli getinfo` 03:23 < m-schmoock> cdecker: will do when t0mix is back 03:26 -!- spinza [~spin@155.93.246.187] has joined #c-lightning 03:34 < t0mix> hmm, maybe that's because new block arrive to my node, it takes good minute+ for my node to validate 03:39 < t0mix> running watch "bitcoin-cli getblockchaininfo|grep blocks; lightning-cli getinfo|grep blockheight" 03:41 <@cdecker> When sending a payment we usually add 1-2 blocks of slack to the last hop, that should counter a slow sync when a new block comes in 03:43 < t0mix> and when I'm not last hop? 03:45 < t0mix> arrived.. "blocks": 577537 vs. "blockheight" : 577536 .. counting seconds 03:45 < t0mix> ~45 seconds 03:46 < t0mix> then sync both at 577537. could that be the problem? 03:47 < t0mix> correction ~50 seconds (with accuracy 2 seconds) 03:48 < t0mix> could that be reason why lnd channels are closed with "sync error" ? 03:51 <@cdecker> That shouldn't affect the lnd channels, that's more to do with the dataloss protection that is acting up a bit (we seem to be counting the commitments differently in some cases) 03:52 < t0mix> I just checked on my other power server. sync was almost immediate after new block arrival. 03:53 < t0mix> why only me is experiencing so many problems with connections to lnd? .. I was almost dancing happy that this must be the cause :( 03:54 <@cdecker> Could be you have a lot of channels? With many attempts there is a higher chance of encountering these issues. 03:54 <@cdecker> I'll fixup my integration tests and try to reproduce the issue 03:54 < t0mix> I have 7, at max I had 22 03:56 < t0mix> thanks cdecker, I'll write you down on my list of people whom I owe a beer. 03:56 <@cdecker> Let's fix it first and then you can do that ^^ 03:57 < t0mix> that'll be more beer 03:57 < m-schmoock> if he'd consume all the beers that are owed to him he'll be drunk all day 03:58 < m-schmoock> t0mix: tell me if/when i should make a test throu your node 03:58 < t0mix> oki, when new block arrive. watching bitcoin debug.log 03:59 < t0mix> once I say now, you can start and continue trying. then let me know when/if successfull 03:59 < m-schmoock> ok 04:00 < m-schmoock> send or receive ? 04:00 < m-schmoock> or anyway? 04:00 < t0mix> anyway 04:00 < t0mix> pick just one 04:00 < m-schmoock> .. im ready 04:02 < t0mix> NOW 04:02 < m-schmoock> started 04:03 < t0mix> aand.. now it is synced 04:04 < m-schmoock> shall we retry next block? 04:04 < m-schmoock> btw payment did not went through yet 04:04 < t0mix> hm :( 04:05 < m-schmoock> this time im getting insufficient fee at 0388a2cd707ac944a2b0c8adbf67c4a313d24b6dc2ab2baba842f9a9eecf925131 04:06 < m-schmoock> or 03a503d8e30f2ff407096d235b5db63b4fcf3f89a653acb6f43d3fc492a7674019 04:06 < m-schmoock> i can try restarting with a fresh gossip store... 04:06 < m-schmoock> let me do this 04:07 < t0mix> ok 04:13 < m-schmoock> so. ready for next block... 04:15 < t0mix> ok.. ready 04:19 < t0mix> NOW 04:20 < m-schmoock> ok 04:20 < m-schmoock> 2 blocks found ... 04:21 < t0mix> synced now 04:21 < t0mix> 70 seconds later 04:21 < t0mix> 2 blocks? I see one 04:21 < t0mix> ah.. now 2 more 04:21 < m-schmoock> first fail 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILUR 04:21 < m-schmoock> (maybe because they all sync now) 04:22 < m-schmoock> so we have to hope that everybody now syncs faster along the route than you 04:22 < t0mix> I believe nobody sync slower than me ='D 04:22 < m-schmoock> lol 04:22 < m-schmoock> t0mix: resolved invcoice 04:22 < m-schmoock> went trhough 04:23 < t0mix> I'm in sync just recently 04:23 < m-schmoock> shall I do more payments? 04:23 < m-schmoock> i just do it... 04:23 < t0mix> you can try now, I'm in sync 04:24 < m-schmoock> resolved again 04:24 < m-schmoock> now i try to withdrew from your channell... 04:24 < t0mix> very good.. try, try 04:27 < m-schmoock> i can see and feel how the network forwarding gets slower when theres blocks in sync... 04:27 < m-schmoock> :D 04:27 < m-schmoock> resolved again.. 04:29 * t0mix put Intel NUC 7i5BNHX1 in the basket.. listing ssd drives.. 04:29 < m-schmoock> ok, are we done with the tests? 04:30 < t0mix> I guess so. now I hope cdecker can come with some more insights. 04:31 < t0mix> thank you for testing +1 beer :) 04:31 < t0mix> I'm out of sync 04:31 * m-schmoock jumping into car heading towards east... 04:31 < t0mix> you can try yet once more 04:31 < m-schmoock> kk 04:32 < t0mix> in sync now 04:33 < t0mix> out now =D 04:34 < t0mix> in sync again 04:34 < m-schmoock> its still trying. 04:35 < t0mix> okay, I think it's enough. 04:36 < m-schmoock> this node is causing a lot of troubles: 0388a2cd707ac944a2b0c8adbf67c4a313d24b6dc2ab2baba842f9a9eecf925131 04:36 < m-schmoock> https://explorer.acinq.co/n/0388a2cd707ac944a2b0c8adbf67c4a313d24b6dc2ab2baba842f9a9eecf925131 04:36 < m-schmoock> it finally went through 04:38 < m-schmoock> im off now... cu l8er 04:38 < t0mix> o/ 05:11 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has joined #c-lightning 05:30 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has quit [Ping timeout: 258 seconds] 05:48 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #c-lightning 06:27 < m-schmoock> cdecker: the mbtc PR is ready for review if you consider it for the next release, else no big deal: https://github.com/ElementsProject/lightning/pull/2657 08:02 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...] 08:03 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has joined #c-lightning 08:09 -!- spinza [~spin@155.93.246.187] has joined #c-lightning 08:12 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has quit [Excess Flood] 08:16 -!- ulrichard [~richi@dzcpe6300borminfo01-e0.static-hfc.datazug.ch] has quit [Remote host closed the connection] 08:17 -!- bitonic-cjp [~bitonic-c@92-111-70-106.static.v4.ziggozakelijk.nl] has quit [Quit: Leaving] 09:12 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has joined #c-lightning 10:02 -!- darosior [6dbe8dc1@gateway/web/freenode/ip.109.190.141.193] has quit [Quit: Page closed] 10:49 -!- Kostenko [~Kostenko@dsl-154-81.bl26.telepac.pt] has quit [Quit: Leaving] 10:57 -!- darosior [52ff9820@gateway/web/freenode/ip.82.255.152.32] has joined #c-lightning 11:23 -!- Kostenko [~Kostenko@176.78.154.81] has joined #c-lightning 11:30 < m-schmoock> Bah, Travis is damn flaky. Third time restard build. .·´¯`(>▂<)´¯`·. 11:37 <@cdecker> Yeah, I just marked a bunch of tests as flaky since that's been going on for a while 11:56 -!- darosior [52ff9820@gateway/web/freenode/ip.82.255.152.32] has quit [Ping timeout: 256 seconds] 12:11 < m-schmoock> I have an old closed channel that is in state "ONCHAIN" for weeks. It still always gets loaded as peer with one log entry: "Loaded from database" 12:11 < m-schmoock> no big deal, just curous 12:11 < m-schmoock> curious 12:20 < roasbeef> t0mix: yeah the other side would close out 12:44 -!- StopAndDecrypt [~StopAndDe@unaffiliated/stopanddecrypt] has joined #c-lightning 12:46 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has joined #c-lightning 13:10 <@cdecker> m-schmoock: might be that we're still tracking some htlcs for that onchain channel 13:15 -!- rh0nj [~rh0nj@88.99.167.175] has quit [Remote host closed the connection] 13:16 -!- rh0nj [~rh0nj@88.99.167.175] has joined #c-lightning 14:01 < jb55> I made a tool https://github.com/jb55/clightning-dumpkeys 14:05 < jb55> because I wanted to add my clightning keys as watchonly addresses to core just for fun + walletnotify alerts. will be a good test of the "combo" output descriptor as well since we share the same pubkey for sh_wpkh and wpkh 14:56 < t0mix> roasbeef: what happen when my node come online? is it checking channel state with its channelled node? what if my node is on block #100, while other node is on block #101 already.. and such state remain for good minute. could that cause some problems? (my cln talking to other lnd) 14:57 -!- copumpkin [~copumpkin@haskell/developer/copumpkin] has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…] 15:30 -!- spinza [~spin@155.93.246.187] has quit [Quit: Coyote finally caught up with me...] 15:42 -!- spinza [~spin@155.93.246.187] has joined #c-lightning 15:43 -!- michaelsdunn1 [~michaelsd@unaffiliated/michaelsdunn1] has quit [Remote host closed the connection] 16:19 -!- darosior [52ff9820@gateway/web/freenode/ip.82.255.152.32] has joined #c-lightning 16:19 < darosior> cdecker: is a "FIXME(cdecker)" reserved for you for some reasons or I can work on the fix ? 16:32 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has quit [Ping timeout: 268 seconds] 16:34 -!- justan0theruser [~justanoth@unaffiliated/justanotheruser] has joined #c-lightning 16:39 -!- justan0theruser [~justanoth@unaffiliated/justanotheruser] has quit [Quit: WeeChat 2.4] 16:40 -!- justanotheruser [~justanoth@unaffiliated/justanotheruser] has joined #c-lightning 16:45 -!- darosior [52ff9820@gateway/web/freenode/ip.82.255.152.32] has quit [Quit: Page closed] 16:55 < roasbeef> t0mix: it'll realize that commitment was closed on chain, and go to sweep it 17:21 -!- Eagle[TM] [~EagleTM@unaffiliated/eagletm] has joined #c-lightning 17:23 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has quit [Ping timeout: 245 seconds] 18:21 -!- Kostenko [~Kostenko@176.78.154.81] has quit [Ping timeout: 248 seconds] 18:23 -!- Kostenko [~Kostenko@176.78.154.81] has joined #c-lightning 19:31 -!- m-schmoock [~will@schmoock.net] has quit [Remote host closed the connection] 20:03 -!- EagleTM [~EagleTM@unaffiliated/eagletm] has joined #c-lightning 20:05 -!- Eagle[TM] [~EagleTM@unaffiliated/eagletm] has quit [Ping timeout: 272 seconds] 20:53 -!- spaced0ut [~spaced0ut@unaffiliated/spaced0ut] has quit [Ping timeout: 248 seconds] 21:10 -!- achow101 [~achow101@unaffiliated/achow101] has quit [Ping timeout: 245 seconds] 21:11 -!- achow101 [~achow101@unaffiliated/achow101] has joined #c-lightning --- Log closed Sat May 25 00:00:22 2019