Agent disconnect after idle exits non-zero

Agent version: v3.33.3

Sadly I don't have a whole lot of diagnostic information but I'm happy to generate more and provide it. The last few months we have been using the self-termination option as documented in several of the posts about more effective scaling policies. We solely scale down our buildkite agent pool by configuring the agent to self-terminate after 25 minutes of idle, and having the systemd hook to remove the agent from the ASG.

Not often, but enough to be annoying, an agent will terminate in the middle of a job. Currently the only records we have of this are in the Buildkite job log which simply reads `Exited with status -1 (agent lost)`.

If there are some basic debug logs we can enable that would capture what's going on here, I'd be all for it.

EDIT: As you can see from my later comments, this is really about the agent exiting non-zero from a disconnection, which triggers systemd to restart the unit - thus causing our early termination when the agent receives another job.

Start by tracing the self-termination and agent-disconnection behavior described in the issue, including the systemd hook that reacts to the agent's exit status. Reproduce or collect debug logs for a disconnected agent, then verify that an ordinary disconnection no longer causes an unintended restart or early termination.

Written by the indexing model from the issue text.