--- Log opened Sun Jan 03 00:00:04 2021
00:17 -!- shesek [~shesek@unaffiliated/shesek] has quit [Remote host closed the connection]
01:30 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has quit [Remote host closed the connection]
01:40 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has joined #c-lightning
02:03 -!- belcher [~belcher@unaffiliated/belcher] has quit [Read error: Connection reset by peer]
02:04 -!- belcher [~belcher@unaffiliated/belcher] has joined #c-lightning
02:10 -!- jasan [~j@94.230.151.172] has joined #c-lightning
02:32 -!- jonatack [~jon@88.124.242.136] has quit [Ping timeout: 260 seconds]
02:34 -!- jonatack [~jon@88.124.242.136] has joined #c-lightning
02:35 -!- jonatack [~jon@88.124.242.136] has quit [Client Quit]
02:40 <@cdecker> Like I mentioned above, the backup is a copy of the database followed by all changes since that copy was taken, so we'd need to regularly re-apply the changes to the DB in order for it to remain small. 
02:41 <@cdecker> Ideally we'd have a backup-rotate function that creates a new backup, safely replacing the old one periodically
02:49 -!- jonatack [~jon@88.124.242.136] has joined #c-lightning
03:00 -!- jonatack [~jon@88.124.242.136] has quit [Ping timeout: 264 seconds]
03:01 -!- jonatack [~jon@213.152.162.99] has joined #c-lightning
03:15 -!- jonatack [~jon@213.152.162.99] has quit [Quit: jonatack]
03:15 -!- jonatack [~jon@213.152.162.99] has joined #c-lightning
03:58 -!- liberliver [~Thunderbi@144.49.211.130.bc.googleusercontent.com] has joined #c-lightning
04:00 -!- liberliver [~Thunderbi@144.49.211.130.bc.googleusercontent.com] has quit [Client Quit]
04:37 < HelloShitty> cdecker: what means exactly to "re-apply the chanages to the DB" ???
04:40 <@cdecker> So you have the original DB, on which changes are applied (SQL statements), and you have the backup file (containing  a copy of the DB and all followup SQL statements). Compaction then just means take the copy, apply all changes, get an up to date version of the DB, use that as the basis for the compacted backup and start appending SQL statements
04:40 <@cdecker> as if it were a new backup
04:41 <@cdecker> "Re-apply" because the changes were already applied to the original, so let's do it on the copy as well
05:28 < HelloShitty> cdecker: isn't the DB the lightningd.sqlight3 itself? I'm sorry, I don't know enough about the database structure nor how is it build
05:29 < HelloShitty> So, I'm struggling to understand how the backup works
05:29 <@cdecker> Yes, I was talking about the backup plugin, which keeps a write-ahead log of all changes
05:30 < HelloShitty> So, this file contains only sql statements about all channel changes, right?
05:30 < HelloShitty> And the first backup command, the 'init' creates and compacts this lightningd.sqlite3 file
05:30 < HelloShitty> Is that it?
05:33 <@cdecker> Sort of: `backup-cli init ldir bpath` creates a file `bpath`, writes a header with metadata and then writes a copy of `lightningd.sqlite3` from `ldir` into the file. The plugin at startup checks that the metadata matches, i.e., we didn't skip any SQL statements which might result in gaps, and then proceeds to interactively append SQL statements as
05:33 <@cdecker> they are applied in lightningd.sqlite3 itself.
05:33 < HelloShitty> hum, ok
05:34 < HelloShitty> So, there are two distinct parts on that backup
05:34 <@cdecker> The backup file therefore is metadata + copy + SQL statements. To get a working DB back we extract the copy, and apply the SQL statements to it until we sync up with the backed up lightningd.sqlite3
05:34 < HelloShitty> ok, but that 'copy' part, also contains sql statements, yes?
05:36 <@cdecker> Not really, the DB is what you get after executing the SQL statements on a DB, but the statements themselves are not stored in the DB (just their results). It's the difference between a diff (SQL statements) and a diff applied to a file (DB)
05:37 < HelloShitty> ok, I see
05:37 < HelloShitty> but I still have one more unanswered question about the backup
05:39 < HelloShitty> So, let's say I did my backup yesterday, and somehow, during the night, I had some problem and a few hours later, I notice  my node is not good, so I need to use the backup to restore my funds
05:40 < HelloShitty> My quesiton is: the backup was only up to date until the node crashed and got problematic. A few hours have passed and now, the backup knows nothing about those missing hours
05:40 < HelloShitty> How is the backup going to work on restoring my funds?
05:40 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]
05:40 < HelloShitty> I mean, how it knows about what happened during those hours the node was down/problematic?
05:40 <@cdecker> If the node itself was offline there is no risk of contradicting ourselves (which our peer would interpret as a cheating attempt).
05:41 <@cdecker> If there are no changes to be backed up, there is nothing to do for the backup plugin in other words
05:41 <@cdecker> I think you might be mistaking backup with a watchtower, which is a completely different thing watching the blockchain for peers that misbehave
05:42 < HelloShitty> hum, ok. I need to learn about those watchtowers then
05:43 < HelloShitty> but about the backup, and in normal conditions, do we need to do anything else after starting the plugin so that the backup is up to date and functional?
05:46 < HelloShitty> but yet about the plugin, if I have a channel open, and that channel was used to forward a payment while my node was problematic (not sure if the problem matters, but let's say the lightningd.sqlite3 files got corrupted)
05:47 < HelloShitty> in this case, can the payment be forward and the channel state/balance not be logged into the database or the payment can't even be forward if it cannot be logged into the database
05:47 < HelloShitty> ?
05:47 < HelloShitty> I'm trying to ask 2 quesitons in 1
05:48 < HelloShitty> Let me try to rephrase
05:48 < HelloShitty> That's conffusing, I guess
05:49 < HelloShitty> Let's say I have a channel open and during the night, it is used to attempt to forward a payment and assuming my lightningd.sqlite3 file is corrupt
05:50 < HelloShitty> can this payment be forward even if my lightnind.sqlite3 file is corrupt? And if it can, this forwarding will not be logged
05:51 < HelloShitty> So, I try to use my backup file to update my node with that forward that happen during the night
05:52 < HelloShitty> Is this a scenatio where a backup can be used? If so, the backup doesn't know about this forwarding either, so how can it be used to recover that payment forwarding and the possible fees my noe collected?
06:09 <@cdecker> Yes, the plugin needs to keep running together with the node, hence the recommendation to use `important-plugin` instead of the `plugin` argument which'd allow the plugin to stop without stopping the node
06:10 <@cdecker> If `lightningd.sqlite3` gets corrupted, then `lightningd` will stop, and the backup will still be in sync (unless it's a silent corruption, in which case there's bigger issues, not just with c-lightning. sqlite3 is pretty good at preventing / detecting them)
06:11 <@cdecker> There is no way I can see in which we'd not be able to recover a payment from the backup, whether it was initiated by us, received by us, or forwarded.
06:12 <@cdecker> If the node is offline however, independently of the cause, it may end up with the channels being settled in order to settle the forwarded payment. The backup will help you recover your funds, but if you don't manually recover the corrupted node before the HTLCs time out, your channels may end up closed.
06:30 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-grtgkzfemgwjcvcq] has left #c-lightning []
06:30 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-grtgkzfemgwjcvcq] has joined #c-lightning
07:08 < nibbier> is there a way to use watchtowers with clighting? in lnd the static channel backups are just a few kilobytes. I assume the clightning backups are better, as they don't require a closing of all channels, but besides this, any other benefits?
07:44 <@cdecker> Yes, you don't rely on your peer to close the channel for you in case of a restore (which is how SCB works)
07:47 <@cdecker> Re watchtower: yes, the guys at the eye of satoshi (TEOS) built their watchtower on top of c-lightning first (https://github.com/talaia-labs/python-teos)
08:00 -!- jasan [~j@94.230.151.172] has quit [Ping timeout: 264 seconds]
08:02 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has quit [Read error: Connection reset by peer]
08:02 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has quit [Read error: Connection reset by peer]
08:02 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Write error: Connection reset by peer]
08:02 -!- mryandao [~mryandao@gateway/tor-sasl/mryandao] has quit [Write error: Connection reset by peer]
08:02 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Write error: Connection reset by peer]
08:02 -!- k3tan [~pi@gateway/tor-sasl/k3tan] has quit [Write error: Connection reset by peer]
08:02 -!- zmnscpxj [~zmnscpxj@gateway/tor-sasl/zmnscpxj] has quit [Read error: Connection reset by peer]
08:02 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has quit [Read error: Connection reset by peer]
08:07 -!- k3tan [~pi@gateway/tor-sasl/k3tan] has joined #c-lightning
08:08 -!- jasan [~j@94.230.151.172] has joined #c-lightning
08:08 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #c-lightning
08:08 -!- bitdex [~bitdex@gateway/tor-sasl/bitdex] has joined #c-lightning
08:08 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has joined #c-lightning
08:09 -!- jb55 [~jb55@gateway/tor-sasl/jb55] has joined #c-lightning
08:09 -!- mryandao [~mryandao@gateway/tor-sasl/mryandao] has joined #c-lightning
08:11 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #c-lightning
09:19 -!- jasan [~j@94.230.151.172] has quit [Ping timeout: 246 seconds]
09:22 < fiatjaf2> how can I debug "no connection to first peer found" when using sendonion?
09:22 < fiatjaf2> what is this connection?
09:22 < fiatjaf2> I can't inject printf statements inside the codebase
09:33 < fiatjaf2> need some basic instructions on how to debug these C structures
09:34 -!- jasan [~j@94.230.151.172] has joined #c-lightning
09:54 -!- alko89 [~alko89@unaffiliated/alko89] has quit [Quit: ZNC 1.7.5 - https://znc.in]
09:59 -!- jasan [~j@94.230.151.172] has quit [Quit: Bye]
09:59 -!- alko89 [~alko89@unaffiliated/alko89] has joined #c-lightning
10:57 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
11:03 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
11:04 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
11:06 < HelloShitty> cdecker: thanks for the explanation. I was away
11:10 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
11:11 < HelloShitty> So, with this plugin running, is it pretty safe to open a larger channel and rest assured that if anything goes wron, I'll still be able to recover any funds in this channel?
11:13 < HelloShitty> I'm running this node on a laptop and the database is in an HDD. It's an external HDD which I think are safer to prevent data loss/corruption. I know SSDs have this kind of time life measured in TBW (Tera bytes written) and after that mark has been taken, data loss/corruption becomes more likely
11:13 < HelloShitty> An as HDDs are mechanic, they might last longer until you start getting problems
11:15 -!- az0re [~az0re@gateway/tor-sasl/az0re] has joined #c-lightning
11:16 < HelloShitty> nibbier: my backup file with 2 days old has already 544Mb
11:16 < HelloShitty> It's growing fast
11:24 < nibbier> HelloShitty: https://github.com/lightningd/plugins/issues/192 - as also mentioned in the chat here, cdecker seems to have a plan on this already.
11:25 < HelloShitty> let me take a look there
11:27 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
11:33 < HelloShitty> I just left my contribuition there
11:33 < HelloShitty> Letting them know that in my case, the backup file is in an ext4 file system which supports files bigger than 2Gb
11:34 < HelloShitty> Let's see if I don't have problems with my node
12:16 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
13:22 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
14:20 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
14:22 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
14:22 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
14:23 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
14:25 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
14:26 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
14:29 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Ping timeout: 240 seconds]
14:31 -!- vasild [~vd@gateway/tor-sasl/vasild] has joined #c-lightning
14:34 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-grtgkzfemgwjcvcq] has left #c-lightning []
14:35 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-grtgkzfemgwjcvcq] has joined #c-lightning
15:56 -!- zmnscpxj [~zmnscpxj@gateway/tor-sasl/zmnscpxj] has joined #c-lightning
16:03 -!- zmnscpxj [~zmnscpxj@gateway/tor-sasl/zmnscpxj] has quit [Remote host closed the connection]
16:28 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
16:29 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
16:30 -!- zmnscpxj [~zmnscpxj@gateway/tor-sasl/zmnscpxj] has joined #c-lightning
16:33 < zmnscpxj> nibbier: just to be clear, watchtowers and backups are two different things. watchtowers cannot help you if your local data storage goes bad. backups cannot help you if your ISP disconnects you because they screwed up.
16:34 < zmnscpxj> (modulo `option_static_pubkey`, watchtowers cannot help you. but you still have to wait if the other side closes, and if the other side *also* loses local storage, the channel is lost forever even with a watchtower)
16:36 < zmnscpxj> HelloShitty: my experience is that HDDs are *less* reliable than SSD, and having an external enclosure means more parts can fail (bumping the enclosure risks disconnecting the cable, etc.)
16:38 < zmnscpxj> and if you're using an extra device anyway, consider using RAID1 of some kind with your laptop storage; without backup-rotate your backup storage usage will just keep growing
17:28 -!- zmnscpxj [~zmnscpxj@gateway/tor-sasl/zmnscpxj] has quit [Remote host closed the connection]
17:28 -!- zmnscpxj [~zmnscpxj@gateway/tor-sasl/zmnscpxj] has joined #c-lightning
17:36 -!- vasild_ [~vd@gateway/tor-sasl/vasild] has joined #c-lightning
17:36 -!- vasild [~vd@gateway/tor-sasl/vasild] has quit [Disconnected by services]
17:36 -!- vasild_ is now known as vasild
17:51 -!- zmnscpxj [~zmnscpxj@gateway/tor-sasl/zmnscpxj] has quit [Ping timeout: 240 seconds]
19:45 -!- ksedgwic [~ksedgwic@192-184-130-48.fiber.dynamic.sonic.net] has joined #c-lightning
21:11 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Remote host closed the connection]
21:11 -!- ghost43 [~daer@gateway/tor-sasl/daer] has joined #c-lightning
21:14 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
21:22 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
22:10 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
22:14 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
22:16 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
22:36 -!- ghost43 [~daer@gateway/tor-sasl/daer] has quit [Quit: Leaving]
22:36 -!- ghost43_ [~daer@gateway/tor-sasl/daer] has joined #c-lightning
22:54 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-grtgkzfemgwjcvcq] has left #c-lightning []
22:55 -!- blockstream_bot [blockstrea@gateway/shell/sameroom/x-grtgkzfemgwjcvcq] has joined #c-lightning
23:07 -!- cryptoso- [~cryptosoa@gateway/tor-sasl/cryptosoap] has joined #c-lightning
23:09 -!- cryptosoap [~cryptosoa@gateway/tor-sasl/cryptosoap] has quit [Ping timeout: 240 seconds]
23:30 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
23:33 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has quit [Remote host closed the connection]
23:41 -!- DeanGuss [~dean@gateway/tor-sasl/deanguss] has joined #c-lightning
--- Log closed Mon Jan 04 00:00:05 2021