nestybox / nestybox/sysbox

sysbox k8s directory mounted as nobody

Open
#800 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
3.9k
Forks
230
Avg merge
7h 48m
Merged PRs (30d)
3

Description

Here is the situation, we are running sysbox in GKE (to run Coder), we have a mount for docker backed by a PVC, sometimes when a pod restarts, /var/lib/docker ens up being owned by nobody:nogroup in the pod:

root@coder:/# ls -lah /var/lib
drwx--x--- 12 nobody nogroup 4.0K May  6 12:30 docker

restarting the pod a bunch of times end up fixing the issue, but not able to figure out why/how
I suspect that this issue happen when the pod gets scheduled in a different node ?

This is quite disruptive as the only way out is to delete that pod and make a new one, loosing the PVC, and the data associated...

pod.yaml
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      coder.workspace_id: e832bafe-2d57-4d56-8e53-a807a86d0869
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        io.kubernetes.cri-o.userns-mode: auto:size=65536
      creationTimestamp: null
      labels:
        coder.workspace_id: e832bafe-2d57-4d56-8e53-a807a86d0869
    spec:
      automountServiceAccountToken: true
      containers:
      - command:
        - sh
        - -c
        - "            set -e\n\n            W_USER=MYUSER\n\n            # Add a
          user so that you're not developing as the `root` user\n            useradd
          $W_USER \\\n              --create-home \\\n              --shell=/bin/bash
          \\\n              --groups=docker \\\n              --uid=1000 \\\n              --user-group\n
          \           echo \"$W_USER ALL=(ALL) NOPASSWD:ALL\" >>/etc/sudoers.d/nopasswd\n\n
          \           # Start the Coder agent as the user once systemd has started
          up\n            # /!\\ The space before EOT must match the current indenting
          of the terminating one!\n            sudo -u $W_USER --preserve-env=CODER_AGENT_TOKEN
          /bin/bash -- <<-'            EOT' &\n            while [[ ! $(systemctl
          is-system-running) =~ ^(running|degraded) ]]\n            do\n              echo
          \"Waiting for system to start... $(systemctl is-system-running)\"\n              sleep
          2\n            done\n            #!/usr/bin/env sh\nset -eux\n# Sleep for
          a good long while before exiting.\n# This is to allow folks to exec into
          a failed workspace and poke around to\n# troubleshoot.\nwaitonexit() {\n\techo
          \"=== Agent script exited with non-zero code. Sleeping 24h to preserve logs...\"\n\tsleep
          86400\n}\ntrap waitonexit EXIT\nBINARY_DIR=\"${BINARY_DIR:-$(mktemp -d -t
          coder.XXXXXX)}\"\nBINARY_NAME=coder\nBINARY_URL=https://coder.company.com/bin/coder-linux-amd64\ncd
          \"$BINARY_DIR\"\n# Attempt to download the coder agent.\n# This could fail
          for a number of reasons, many of which are likely transient.\n# So just
          keep trying!\nwhile :; do\n\t# Try a number of different download tools,
          as we don not know what we\n\t# will have available.\n\tstatus=\"\"\n\tif
          command -v curl >/dev/null 2>&1; then\n\t\tcurl -fsSL --compressed \"${BINARY_URL}\"
          -o \"${BINARY_NAME}\" && break\n\t\tstatus=$?\n\telif command -v wget >/dev/null
          2>&1; then\n\t\twget -q \"${BINARY_URL}\" -O \"${BINARY_NAME}\" && break\n\t\tstatus=$?\n\telif
          command -v busybox >/dev/null 2>&1; then\n\t\tbusybox wget -q \"${BINARY_URL}\"
          -O \"${BINARY_NAME}\" && break\n\t\tstatus=$?\n\telse\n\t\techo \"error:
          no download tool found, please install curl, wget or busybox wget\"\n\t\texit
          127\n\tfi\n\techo \"error: failed to download coder agent\"\n\techo \"       command
          returned: ${status}\"\n\techo \"Trying again in 30 seconds...\"\n\tsleep
          30\ndone\n\nif ! chmod +x $BINARY_NAME; then\n\techo \"Failed to make $BINARY_NAME
          executable\"\n\texit 1\nfi\n\nhaslibcap2() {\n\tcommand -v setcap /dev/null
          2>&1\n\tcommand -v capsh /dev/null 2>&1\n}\nprintnetadminmissing() {\n\techo
          \"The root user does not have CAP_NET_ADMIN permission. \" + \\\n\t\t\"If
          running in Docker, add the capability to the container for \" + \\\n\t\t\"improved
          network performance.\"\n\techo \"This has security implications. See https://man7.org/linux/man-pages/man7/capabilities.7.html\"\n}\n\n#
          Attempt to add CAP_NET_ADMIN to the agent binary. This allows us to increase\n#
          network buffers which improves network transfer speeds.\nif [ -n \"${USE_CAP_NET_ADMIN:-}\"
          ]; then\n\t# If running as root, we do not need to do anything.\n\tif [
          \"$(id -u)\" -eq 0 ]; then\n\t\techo \"Running as root, skipping setcap\"\n\t\t#
          Warn the user if root does not have CAP_NET_ADMIN.\n\t\tif ! capsh --has-p=CAP_NET_ADMIN;
          then\n\t\t\tprintnetadminmissing\n\t\tfi\n\n\t# If not running as root,
          make sure we have sudo perms and the \"setcap\" +\n\t# \"capsh\" binaries
          exist.\n\telif sudo -nl && haslibcap2; then\n\t\t# Make sure the root user
          has CAP_NET_ADMIN.\n\t\tif sudo -n capsh --has-p=CAP_NET_ADMIN; then\n\t\t\tsudo
          -n setcap CAP_NET_ADMIN=+ep ./$BINARY_NAME || true\n\t\telse\n\t\t\tprintnetadminmissing\n\t\tfi\n\n\t#
          If we are not running as root, cant sudo, and \"setcap\" does not exist,
          we\n\t# cannot do anything.\n\telse\n\t\techo \"Unable to setcap agent binary.
          To enable improved network performance, \" + \\\n\t\t\t\"give the agent
          passwordless sudo permissions and the \\\"setcap\\\" + \\\"capsh\\\" binaries.\"\n\t\techo
          \"This has security implications. See https://man7.org/linux/man-pages/man7/capabilities.7.html\"\n\tfi\nfi\n\nexport
          CODER_AGENT_AUTH=\"token\"\nexport CODER_AGENT_URL=\"https://coder.company.com/\"\nexec
          ./$BINARY_NAME agent\n\n            EOT\n\n            exec /sbin/init\n"
        env:
        - name: CODER_AGENT_TOKEN
          value: XXXXX
        - name: SYSBOX_ALLOW_TRUSTED_XATTR
          value: "FALSE"
        image: us.gcr.io/XXX/docker-image-systemd
        imagePullPolicy: IfNotPresent
        name: coder-MYUSER-0
        resources:
          limits:
            cpu: "1"
            memory: 4Gi
          requests:
            cpu: "1"
            memory: 4Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /ws
          mountPropagation: None
          name: data
          subPath: workspaces
        - mountPath: /home
          mountPropagation: None
          name: data
          subPath: home
        - mountPath: /var/lib/docker
          mountPropagation: None
          name: data
          subPath: var/lib/docker
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      hostname: coder-MYUSER-0
      restartPolicy: Always
      runtimeClassName: sysbox-runc
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: OnRootMismatch
        runAsNonRoot: false
        runAsUser: 0
      shareProcessNamespace: false
      terminationGracePeriodSeconds: 30
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: coder-e832bafe-2d57-4d56-8e53-a807a86d0869

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the embedded pod.yaml, focusing on the PVC subPath mount at /var/lib/docker, fsGroup settings, and the sysbox-runc runtime. Reproduce the ownership change across pod restarts or node rescheduling and compare the mount state; done means identifying the cause and a safe way to preserve the PVC data and expected ownership.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, gcp, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.