Recover from disconnects

A couple times in the last few weeks mockturtle has lost its connection to Libera. It continues running but can’t reconnect.

Maybe the straightforward fix here is to detect the net connection has dropped and exit the process? I’d rather not build complicated reconnect logic, let a new process rebuild. It’s not so bad if we lose a single post along the way, though depending when and where the first exception happens, maybe the cache file won’t be updated so the new process will handle it?

Sep 16 19:25:28 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":false},"message":"Activating scheduled tasks.","level":"info","timestamp":"2024-09-16 19:25:28"}
Sep 16 19:29:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":false},"message":"Activating scheduled tasks.","level":"info","timestamp":"2024-09-16 19:29:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Client Ping timeout (120 seconds)","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.end() connected=true with data=true had_error=true","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.end() connected=true with data=false had_error=true","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport close() destroying","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketClose()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket closed. was_connected=true safely_registered=true requested_disconnect=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Scheduling reconnect. Attempt: 1/3 Wait: 6894ms","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.setEncoding() encoding=utf8","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport connect()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport disposeSocket() connected=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport Connection.setEncoding() encoding=utf8","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport Connecting socket..","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketRawConnected()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:54 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketFullyConnected()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:54"}
Sep 16 19:31:54 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket fully connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:54"}
Sep 16 19:32:27 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketClose()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:32:27"}
Sep 16 19:32:27 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket closed. was_connected=true safely_registered=false requested_disconnect=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:32:27"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":"mockturtle"},"itemDate":"2024-09-16T21:21:01.000Z","itemGuid":"https://lobste.rs/s/mun2dn","lastSeen":"2024-09-16T21:21:01.000Z","level":"info","message":"Broadcasting story","timestamp":"2024-09-16 21:21:37"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"write() called when not connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 21:21:37"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"write() called when not connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 21:21:37"}

No files or tests are named. Start by tracing the IRC connection lifecycle around the logged ping timeout, reconnect attempts, and later writes while disconnected, then inspect how cache updates are sequenced. Done means the selected disconnect behavior is covered by a reproducible test and the bot no longer remains unusable after losing its connection.