Connection between worker & channel closes, can't reconnect (lost runs?)

Connections between ws-worker and lightning getting closed/dropped and somehow not reconnecting.

We don't really know how to reproduce or fix this yet, but if it happens a run will be lost. We have not seen any runs lost related to this in the last 6 days, so it's possible that it's been resolved by other changes but we want to keep this issue open for a little while just in case

cc @josephjclark , @rorymckinley , @stuartc , @theroinaochieng

Details on Slack

• @Joe This is in the logs before the timeout: Critical error in channel run:ed9d8f83-65f8-4721-8132-814303606b94
• @Rory that's an error coming out of the phoenix socket - means it can't talk on the channel. No error details returned. Annoying. So yes, looks like a connectivity issue on the socket. And after that point other requests start failing

Start by reviewing the ws-worker/lightning channel handling and the Phoenix socket error described in the issue, then consult the linked Slack thread and the recorded critical-error log. The issue needs a reproducible closed-connection case before a fix can be defined; done should include reliable reconnection without lost runs and evidence covering the failure path.