actions / actions/actions-runner-controller

cgroupv2 is not respecting dockerdContainerResources

Open
#2,284 21 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: ghrunner
spec:
  replicas: 1
  template:
    spec:
      dockerdContainerResources:
        limits:
          cpu: "4"
          memory: 1000Mi
        requests:
          cpu: 50m
          memory: 1000Mi
      ephemeral: true
      image: ""
      labels:
      - Linux
      - X64
      - self-hosted
      - mylabel
      organization: xyz
      resources:
        limits:
          cpu: "4"
          memory: 800Mi
        requests:
          cpu: 50m
          memory: 800Mi
To Reproduce

Execute following in a workflow or directly on the runner:

docker info // make sure it has "Cgroup Version: 2" and "Cgroup Driver: cgroupfs"*
docker run -it ubuntu bash
cat /dev/zero | head -c 2000000000 | tail

*which is the default, when dind is started on a kind v0.17.0 (k8s v1.25.3) cluster on Ubuntu 22.04.1 LTS.

Describe the bug

You can consume more memory than in the limits specified.
And kubectl top <runner pod> is missing the memory/cpu of the nested containers.

Describe the expected behavior

The dind container/process should have been killed.
(this works fine if the node has support for cgroup v1, e.g. with kind on ubuntu 18)

EDIT: killing works fine again with newer Docker versions, but kubectl top still shows the wrong data. Please see linked issues.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the workflow on kind v0.17.0 with Kubernetes 1.25.3, Ubuntu 22.04.1, cgroupv2, and the cgroupfs driver; start by checking docker info and kubectl top for the runner pod. Trace how dockerdContainerResources are applied to nested containers. Done means nested container CPU and memory are enforced and included in the runner pod metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes
Domain
infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.