Call a run "lost" if the socket is closed for N minutes

Run Timeouts Today

Right now, Lightning will mark a run Lost if it does not send run:complete within N minutes after a run is claimed. I think it's 5 minutes? Whatever.

This is like an absolute timeout on the run - the whole thing must be done in this time. The worker tracks its own timeout per run.

Lost Runs

I think the only time Lost occurs is a) if the Worker and Lightning lose connectivity for a long time; or b) if the Worker main process dies before run:complete could be sent.

Lightning should trust the worker to handle any errors which may occur and return an run:complete. It's only a "catastrophic" failure that should cause a Lost state - and they should be very very rare (getting more rare as the service and deployment matures).

Proposal

I think a more useful vision of the Lost timeout is: if the worker's connection to the run socket goes away for N minutes, write the run off as lost.

This ensures that each run gets its full timeout duration to execute. If the run fails or takes too long, the worker will kill it and return to Lightning.

The Phoenix socket connection should I think know that a client is connected, and throw an error or timeout if the client goes away unexpectedly. This would only happen in the event of a catastrophic bug, like a worker crash or internet failure.

So I think the solution is as simple as: set a timeout on the socket connection, and if it fail to reconnect after that timeout, mark the run as Lost.

Start by tracing the Phoenix socket connection lifecycle and the existing run Lost timeout behavior. Confirm how worker disconnections and reconnections are represented, then define the timeout that should mark a run Lost while preserving the worker's own execution timeout. Done means an unexpectedly disconnected socket leads to Lost only after the configured interval.