Jobs hangs if container restarts
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
I am running self-hosted github runners on my k8s env. I had one job which was running on container github-runner-java-3-7v8gr-0, after sometime for some reason the container restarted. I see logs of that time
"A runner exists with the same name"
--
| | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.577694610Z\" level=info msg=\"Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address\""
| | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.604146844Z\" level=info msg=\"Loading containers: done.\""
| | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.614775844Z\" level=info msg=\"Docker daemon\" commit=03df974 graphdriver(s)=overlay2 version=20.10.20"
| | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.614921603Z\" level=info msg=\"Daemon has completed initialization\""
| | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.629788330Z\" level=info msg=\"API listen on /var/run/docker.sock\""
| | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.636562738Z\" level=info msg=\"API listen on [::]:2376\""
| | 2023-02-17 08:16:33 | message="√ Successfully replaced the runner"
| | 2023-02-17 08:16:34 | message="√ Runner connection is good"
| | 2023-02-17 08:16:34 | message=""
| | 2023-02-17 08:16:34 | message="# Runner settings"
| | 2023-02-17 08:16:34 | message=""
| | 2023-02-17 08:16:34 | message=""
| | 2023-02-17 08:16:34 | message="√ Settings Saved."
| | 2023-02-17 08:16:34 | message=""
| | 2023-02-17 08:16:34 | message="\u001b[0;37m2023-02-17 02:46:34.744 DEBUG --- Runner successfully configured.\u001b[0m"
| | 2023-02-17 08:16:34 | message="{"
| | 2023-02-17 08:16:34 | message=" \"agentId\": 17696,"
| | 2023-02-17 08:16:34 | message=" \"agentName\": \"github-runner-java-3-7v8gr-0\","
| | 2023-02-17 08:16:34 | message=" \"poolId\": 1,"
| | 2023-02-17 08:16:34 | message=" \"poolName\": \"Default\","
| | 2023-02-17 08:16:34 | message=" \"ephemeral\": true,"
| | 2023-02-17 08:16:34 | message=" \"serverUrl\": \"https://pipelines.actions.githubusercontent.com/0Wps8TVRrGNT6Sws04E0EqoYQUCG3hwv5brlpw4i2xj7tR7Tty\","
| | 2023-02-17 08:16:34 | message=" \"gitHubUrl\": \"https://github.com/softwareartistry\","
| | 2023-02-17 08:16:34 | message=" \"workFolder\": \"/runner/_work\""
| | 2023-02-17 08:16:34 | message="\u001b[0;37m2023-02-17 02:46:34.749 DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled\u001b[0m"
| | 2023-02-17 08:16:34 | message="\u001b[0;37m2023-02-17 02:46:34.750 DEBUG --- Waiting until Docker is available or the timeout is reached\u001b[0m"
| | 2023-02-17 08:16:34 | message="}CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES"
| | 2023-02-17 08:16:36 | message=""
| | 2023-02-17 08:16:36 | message="√ Connected to GitHub"
| | 2023-02-17 08:16:36 | message=""
| | 2023-02-17 08:16:36 | message="Current runner version: '2.299.1'"
| | 2023-02-17 08:16:36 | message="2023-02-17 02:46:36Z: Listening for Jobs"
So this container has started listening for a new job but the job which was running here is in a hang state now.
To Reproduce
Steps to reproduce the behaviour:
Not sure how to reproduce
Expected behaviour
The job should continue running on new container
Runner Version and Platform
Version of your runner?
2.299.1
OS of the machine running the runner? OSX/Windows/Linux/...
Linux , runner running on k8s
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the runner lifecycle around container restarts and the reported logs showing “Successfully replaced the runner” followed by “Listening for Jobs.” Compare the original job state with the restarted runner’s registration and job-handling behavior in a Kubernetes environment. Done means a restarted container does not leave the original job hanging, or the failure is clearly documented with a tested recovery path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, kubernetes, linux
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100