actions / actions/runner-container-hooks

Multiple issues after upgrade to v0.8.0

Open
#266 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
153
Forks
112
Avg merge
6m
Merged PRs (30d)
1

Description

We are using github enterprise on prem with runner groups running on kubernetes using actions-runner-controller. We are not using the official runner images i.e. ghcr.io/actions/actions-runner:latest but instead are building our own using fedora as base image.

I understand that this may not be the way you expect this to be used but so far it worked really well. Our kubernetes cluster doesn't have access to the internet so we installed all the build dependencies inside the runner image to work around this limitation.

Most of our builds are running inside this runner image juste fine, only a few are using actions which spawn additional containers.

After we upgraded the container hooks to v0.8.0 we had multiple issues. We managed to work around some but in the end had to rollback to v0.7.0

Below is a snipper of the pod that is created during the build :

initContainers:
    - command:
        - bash
        - -c
        - sudo cp $(which sh) /mnt/externals/sh         && sudo cp $(which tail)
          /mnt/externals/tail         && sudo cp $(which env)
          /mnt/externals/env         && sudo chmod -R 777 /mnt/externals
      name: fs-init
      securityContext:
        privileged: true
        runAsGroup: 1001
        runAsUser: 1001

As you can see the initContainer requires sudo but we don't typically allow our user in the runner to run anything as sudo. Despite our concerns we decided to make the change in our runner images to move ahead.

We also couldn't find a way to change the securityContext for these pods so we had to make some more changes to the runner for this to work.

In the end the initContainer managed to copy the files into the volume.

After that the actual container was failing. In this particular case we are building a dotnet app using mcr.microsoft.com/dotnet/sdk:8.0 from Microsoft. Below is the exact error we got during the build :

/__e/tail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /__e/tail)

I believe you are using debian as base for the official runner images, maybe tail on debian doesn't need the same libraries. I also have no idea what microsoft has installed in the dotnet image or what base image they are using.

Lets imagine that this step uses a distroless container e.g. a golang executable using scratch as base image. Would this technique still work ? Maybe if sh, env and tail are statically linked it could but does this even exists ?

At this point I am in way above my knowledge and expertise so we gave up and rolled back to v0.7.0 which worked fine.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the v0.8.0 and v0.7.0 container-hook behavior for the Kubernetes pod shown, including the fs-init initContainer and copied sh, tail, and env binaries. Reproduce with the custom Fedora-based runner and the mcr.microsoft.com/dotnet/sdk:8.0 container, then determine whether the hook can support differing libc versions, securityContext settings, and distroless images.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, kubernetes, typescript
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.