"out of disk" error wedges ephemeral invocations

Describe the bug
In production, we see one-shot github actions runner invocations never return if they hit an out-of-disk condition.

To Reproduce
see claude analysis below.

Expected behavior

exit even if there are I/O errors.

Runner Version and Platform

2.336.0 - latest version

linux x64.

What's not working?

Claude:

Ephemeral runner fails to exit when the disk is full

Summary

An ephemeral (single-shot) runner is expected to run one job and then exit so the
orchestrator can tear down the VM. When the disk fills up, this does not happen:
the runner's own diagnostic logging throws IOException, and in the
Runner.Listener process (unlike Runner.Worker) those trace calls are
unguarded. An uncaught IOException on the exit path is mapped to exit code 2
(RetryableError), which the run.sh wrapper unconditionally treats as
"restart". The runner is relaunched in a loop instead of terminating.

Intended single-shot exit path

• src/Runner.Listener/Runner.cs:331 — ephemeral / --once sets runOnce = true.

• After a job is dispatched, runOnceJobReceived = true
  (src/Runner.Listener/Runner.cs:768).

• On the next loop iteration the runner races the message poll against job
  completion (src/Runner.Listener/Runner.cs:571):

  Task completeTask = await Task.WhenAny(getNextMessage, jobDispatcher.RunOnceJobCompleted.Task);
  if (completeTask == jobDispatcher.RunOnceJobCompleted.Task)
  {
      runOnceJobCompleted = true;
      Trace.Info("Job has finished at backend, the runner will exit ...");  // (a)
      Trace.Info("Stop message queue looping.");                            // (b)
      messageQueueLoopTokenSource.Cancel();
      ...
      return Constants.Runner.ReturnCode.Success;                           // (c)
  }
  

• RunOnceJobCompleted is set in a finally
  (src/Runner.Listener/JobDispatcher.cs:352-353), so it always fires — that part
  is robust.

Where disk-full breaks it

Every Trace.Info/Error/... call funnels through HostTraceListener.WriteLine,
which calls Flush() on every single line
(src/Runner.Common/HostTraceListener.cs:79-91). When the volume is full,
Flush() throws IOException.

In the Worker this is explicitly anticipated and swallowed
(src/Runner.Worker/Program.cs:62-67):

catch (Exception e)
{
    // make sure we don't crash the app on trace error.
    // since IOException will throw when we run out of disk space.
    Console.WriteLine(e.ToString());
}

But in the Listener there is no such guard around the trace calls in the exit
path. Lines (a)/(b) above throw IOException, so the return Success at (c)
never executes. The exception propagates out of the message loop into the outer
finally (src/Runner.Listener/Runner.cs:852-881) — which itself does more
tracing and network I/O (DeleteSessionAsync, and crucially
configManager.DeleteLocalRunnerConfig() at line 879). If any of those
trace/flush calls throw again, the finally aborts before the config cleanup
runs.

The exception then reaches the top-level handler
(src/Runner.Listener/Program.cs:154-158):

catch (Exception e)   // IOException lands here
{
    ...
    return Constants.Runner.ReturnCode.RetryableError;   // exit code 2
}

Why exit code 2 means "does not exit"

run.sh / run-helper.sh have no concept of ephemeral. Exit code 2 is
unconditionally treated as "restart":

• src/Misc/layoutroot/run-helper.sh.template:47-50 → sleeps 5s, exits 2.
• src/Misc/layoutroot/run.sh:20-21 → returnCode -eq 2 → "Restarting runner..."
  → loops forever.

So on a full disk the ephemeral runner keeps getting relaunched instead of
terminating. Two sub-cases:

• Config cleanup was skipped (the finally threw before line 879): the runner
  is still "configured", recreates a session, polls, and every trace/flush hits
  the full disk again → exit 2 → restart → repeat. This is the persistent
  "won't exit" symptom.
• Even if cleanup did run, the intended single-shot semantics (exit and let the
  orchestrator tear down the VM) are violated by the restart in between.

Contributing factors

• Asymmetry: the code base clearly knows disk-full manifests as IOException
  — it is handled in the Worker (src/Runner.Worker/Program.cs:65), in
  src/Runner.Listener/JobDispatcher.cs:566-571 (force-fail the job when the
  worker's stderr contains IOException), and surfaced as a user warning in
  src/Runner.Worker/JobExtension.cs:1156. The Listener's own exit/cleanup path
  was not hardened the same way.
• Every trace line flushes (src/Runner.Common/HostTraceListener.cs:91), so
  there is no buffering slack — the first log line after the disk fills throws.
• The wrapper cannot distinguish a transient retryable error from a fatal
  environmental one, and ephemeral runners still go through the restart loop.

Suggested fixes

1. Wrap Trace calls in the ephemeral exit/cleanup path — or better, make
   HostTraceListener.WriteLine/Flush swallow IOException like the Worker
   already does — so a full disk cannot convert a Success return into exit
   code 2.
2. Ensure DeleteLocalRunnerConfig() for settings.Ephemeral && runOnceJobCompleted
   runs even if earlier finally statements throw (guard each step).
3. Give ephemeral runners a distinct fatal exit code (or have run-helper.sh not
   restart when the local config is already gone / when ephemeral), so exit
   code 2 does not loop.

The smallest change that closes the root cause is hardening HostTraceListener
against IOException.

Start with src/Runner.Common/HostTraceListener.cs and compare its WriteLine/Flush behavior with the exception handling in src/Runner.Worker/Program.cs. Trace the ephemeral completion path in src/Runner.Listener/Runner.cs and verify that an IOException during logging cannot prevent successful exit or local configuration cleanup.