Ephemeral runner hangs indefinitely after GitHub outage and does not recover
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
We spawn ephemeral runners for each github job we receive.
During recent outage, ephemeral runners got stuck not picking up jobs even though
they were registered with github and printed listening for jobs.
there is no way for us to tell if the runner is just waiting for a job, or it's genuinely stuck, so it's not safe
for us to kill due to a small chance that it picks up a job just before we kill it.
Basically. Once a runner is registered and listening, there is no safe way for us to shut it down without race conditions
To Reproduce
- Have a GitHub Outage
- Wait for outage to recover
- Notice all our runners are stuck
Expected behavior
The runner should eventually exit with a non-zero exit code if it fails to pick up jobs. Would even help for us if there is an --exit-after-idle <hours-idle> flag or something.
Runner Version and Platform
Version of your runner?
OS of the machine running the runner? OSX/Windows/Linux/...
What's not working?
Please include error messages and screenshots.
Job Log Output
If applicable, include the relevant part of the job / step log output here. All sensitive information should already be masked out, but please double-check before pasting here.
Runner and Worker's Diagnostic Logs
If applicable, add relevant diagnostic log information. Logs are located in the runner's _diag folder. The runner logs are prefixed with Runner_ and the worker logs are prefixed with Worker_. Each job run correlates to a worker log. All sensitive information should already be masked out, but please double-check before pasting here.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the runner's registration and job-listening behavior during outage recovery, then inspect the _diag files mentioned in the issue, especially Runner_ and Worker_ logs. Done would mean defining and validating a safe recovery or idle-exit behavior that prevents an ephemeral runner from hanging indefinitely and provides a non-zero exit when it cannot pick up work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100