actions / actions/runner

Ephemeral runner hangs indefinitely after GitHub outage and does not recover

Open
#4,076 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug

We spawn ephemeral runners for each github job we receive.
During recent outage, ephemeral runners got stuck not picking up jobs even though
they were registered with github and printed listening for jobs.

there is no way for us to tell if the runner is just waiting for a job, or it's genuinely stuck, so it's not safe
for us to kill due to a small chance that it picks up a job just before we kill it.

Basically. Once a runner is registered and listening, there is no safe way for us to shut it down without race conditions

To Reproduce

  1. Have a GitHub Outage
  2. Wait for outage to recover
  3. Notice all our runners are stuck

Expected behavior

The runner should eventually exit with a non-zero exit code if it fails to pick up jobs. Would even help for us if there is an --exit-after-idle <hours-idle> flag or something.

Runner Version and Platform

Version of your runner?

OS of the machine running the runner? OSX/Windows/Linux/...

What's not working?

Please include error messages and screenshots.

Job Log Output

If applicable, include the relevant part of the job / step log output here. All sensitive information should already be masked out, but please double-check before pasting here.

Runner and Worker's Diagnostic Logs

If applicable, add relevant diagnostic log information. Logs are located in the runner's _diag folder. The runner logs are prefixed with Runner_ and the worker logs are prefixed with Worker_. Each job run correlates to a worker log. All sensitive information should already be masked out, but please double-check before pasting here.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the runner's registration and job-listening behavior during outage recovery, then inspect the _diag files mentioned in the issue, especially Runner_ and Worker_ logs. Done would mean defining and validating a safe recovery or idle-exit behavior that prevents an ephemeral runner from hanging indefinitely and provides a non-zero exit when it cannot pick up work.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.