actions / actions/runner

Jobs hangs if container restarts

Open
#2,446 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
I am running self-hosted github runners on my k8s env. I had one job which was running on container github-runner-java-3-7v8gr-0, after sometime for some reason the container restarted. I see logs of that time

"A runner exists with the same name"
--
  |   | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.577694610Z\" level=info msg=\"Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address\""
  |   | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.604146844Z\" level=info msg=\"Loading containers: done.\""
  |   | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.614775844Z\" level=info msg=\"Docker daemon\" commit=03df974 graphdriver(s)=overlay2 version=20.10.20"
  |   | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.614921603Z\" level=info msg=\"Daemon has completed initialization\""
  |   | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.629788330Z\" level=info msg=\"API listen on /var/run/docker.sock\""
  |   | 2023-02-17 08:16:33 | message="time=\"2023-02-17T02:46:33.636562738Z\" level=info msg=\"API listen on [::]:2376\""
  |   | 2023-02-17 08:16:33 | message="√ Successfully replaced the runner"
  |   | 2023-02-17 08:16:34 | message="√ Runner connection is good"
  |   | 2023-02-17 08:16:34 | message=""
  |   | 2023-02-17 08:16:34 | message="# Runner settings"
  |   | 2023-02-17 08:16:34 | message=""
  |   | 2023-02-17 08:16:34 | message=""
  |   | 2023-02-17 08:16:34 | message="√ Settings Saved."
  |   | 2023-02-17 08:16:34 | message=""
  |   | 2023-02-17 08:16:34 | message="\u001b[0;37m2023-02-17 02:46:34.744  DEBUG --- Runner successfully configured.\u001b[0m"
  |   | 2023-02-17 08:16:34 | message="{"
  |   | 2023-02-17 08:16:34 | message="  \"agentId\": 17696,"
  |   | 2023-02-17 08:16:34 | message="  \"agentName\": \"github-runner-java-3-7v8gr-0\","
  |   | 2023-02-17 08:16:34 | message="  \"poolId\": 1,"
  |   | 2023-02-17 08:16:34 | message="  \"poolName\": \"Default\","
  |   | 2023-02-17 08:16:34 | message="  \"ephemeral\": true,"
  |   | 2023-02-17 08:16:34 | message="  \"serverUrl\": \"https://pipelines.actions.githubusercontent.com/0Wps8TVRrGNT6Sws04E0EqoYQUCG3hwv5brlpw4i2xj7tR7Tty\","
  |   | 2023-02-17 08:16:34 | message="  \"gitHubUrl\": \"https://github.com/softwareartistry\","
  |   | 2023-02-17 08:16:34 | message="  \"workFolder\": \"/runner/_work\""
  |   | 2023-02-17 08:16:34 | message="\u001b[0;37m2023-02-17 02:46:34.749  DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled\u001b[0m"
  |   | 2023-02-17 08:16:34 | message="\u001b[0;37m2023-02-17 02:46:34.750  DEBUG --- Waiting until Docker is available or the timeout is reached\u001b[0m"
  |   | 2023-02-17 08:16:34 | message="}CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES"
  |   | 2023-02-17 08:16:36 | message=""
  |   | 2023-02-17 08:16:36 | message="√ Connected to GitHub"
  |   | 2023-02-17 08:16:36 | message=""
  |   | 2023-02-17 08:16:36 | message="Current runner version: '2.299.1'"
  |   | 2023-02-17 08:16:36 | message="2023-02-17 02:46:36Z: Listening for Jobs"

So this container has started listening for a new job but the job which was running here is in a hang state now.

image

To Reproduce
Steps to reproduce the behaviour:
Not sure how to reproduce

Expected behaviour
The job should continue running on new container

Runner Version and Platform

Version of your runner?
2.299.1

OS of the machine running the runner? OSX/Windows/Linux/...
Linux , runner running on k8s

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the runner lifecycle around container restarts and the reported logs showing “Successfully replaced the runner” followed by “Listening for Jobs.” Compare the original job state with the restarted runner’s registration and job-handling behavior in a Kubernetes environment. Done means a restarted container does not leave the original job hanging, or the failure is clearly documented with a tested recovery path.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, kubernetes, linux
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.