sysbox k8s directory mounted as nobody
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 3.9k
- Forks
- 230
- Avg merge
- 7h 48m
- Merged PRs (30d)
- 3
Description
Here is the situation, we are running sysbox in GKE (to run Coder), we have a mount for docker backed by a PVC, sometimes when a pod restarts, /var/lib/docker ens up being owned by nobody:nogroup in the pod:
root@coder:/# ls -lah /var/lib
drwx--x--- 12 nobody nogroup 4.0K May 6 12:30 docker
restarting the pod a bunch of times end up fixing the issue, but not able to figure out why/how
I suspect that this issue happen when the pod gets scheduled in a different node ?
This is quite disruptive as the only way out is to delete that pod and make a new one, loosing the PVC, and the data associated...
pod.yaml
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
coder.workspace_id: e832bafe-2d57-4d56-8e53-a807a86d0869
strategy:
type: Recreate
template:
metadata:
annotations:
io.kubernetes.cri-o.userns-mode: auto:size=65536
creationTimestamp: null
labels:
coder.workspace_id: e832bafe-2d57-4d56-8e53-a807a86d0869
spec:
automountServiceAccountToken: true
containers:
- command:
- sh
- -c
- " set -e\n\n W_USER=MYUSER\n\n # Add a
user so that you're not developing as the `root` user\n useradd
$W_USER \\\n --create-home \\\n --shell=/bin/bash
\\\n --groups=docker \\\n --uid=1000 \\\n --user-group\n
\ echo \"$W_USER ALL=(ALL) NOPASSWD:ALL\" >>/etc/sudoers.d/nopasswd\n\n
\ # Start the Coder agent as the user once systemd has started
up\n # /!\\ The space before EOT must match the current indenting
of the terminating one!\n sudo -u $W_USER --preserve-env=CODER_AGENT_TOKEN
/bin/bash -- <<-' EOT' &\n while [[ ! $(systemctl
is-system-running) =~ ^(running|degraded) ]]\n do\n echo
\"Waiting for system to start... $(systemctl is-system-running)\"\n sleep
2\n done\n #!/usr/bin/env sh\nset -eux\n# Sleep for
a good long while before exiting.\n# This is to allow folks to exec into
a failed workspace and poke around to\n# troubleshoot.\nwaitonexit() {\n\techo
\"=== Agent script exited with non-zero code. Sleeping 24h to preserve logs...\"\n\tsleep
86400\n}\ntrap waitonexit EXIT\nBINARY_DIR=\"${BINARY_DIR:-$(mktemp -d -t
coder.XXXXXX)}\"\nBINARY_NAME=coder\nBINARY_URL=https://coder.company.com/bin/coder-linux-amd64\ncd
\"$BINARY_DIR\"\n# Attempt to download the coder agent.\n# This could fail
for a number of reasons, many of which are likely transient.\n# So just
keep trying!\nwhile :; do\n\t# Try a number of different download tools,
as we don not know what we\n\t# will have available.\n\tstatus=\"\"\n\tif
command -v curl >/dev/null 2>&1; then\n\t\tcurl -fsSL --compressed \"${BINARY_URL}\"
-o \"${BINARY_NAME}\" && break\n\t\tstatus=$?\n\telif command -v wget >/dev/null
2>&1; then\n\t\twget -q \"${BINARY_URL}\" -O \"${BINARY_NAME}\" && break\n\t\tstatus=$?\n\telif
command -v busybox >/dev/null 2>&1; then\n\t\tbusybox wget -q \"${BINARY_URL}\"
-O \"${BINARY_NAME}\" && break\n\t\tstatus=$?\n\telse\n\t\techo \"error:
no download tool found, please install curl, wget or busybox wget\"\n\t\texit
127\n\tfi\n\techo \"error: failed to download coder agent\"\n\techo \" command
returned: ${status}\"\n\techo \"Trying again in 30 seconds...\"\n\tsleep
30\ndone\n\nif ! chmod +x $BINARY_NAME; then\n\techo \"Failed to make $BINARY_NAME
executable\"\n\texit 1\nfi\n\nhaslibcap2() {\n\tcommand -v setcap /dev/null
2>&1\n\tcommand -v capsh /dev/null 2>&1\n}\nprintnetadminmissing() {\n\techo
\"The root user does not have CAP_NET_ADMIN permission. \" + \\\n\t\t\"If
running in Docker, add the capability to the container for \" + \\\n\t\t\"improved
network performance.\"\n\techo \"This has security implications. See https://man7.org/linux/man-pages/man7/capabilities.7.html\"\n}\n\n#
Attempt to add CAP_NET_ADMIN to the agent binary. This allows us to increase\n#
network buffers which improves network transfer speeds.\nif [ -n \"${USE_CAP_NET_ADMIN:-}\"
]; then\n\t# If running as root, we do not need to do anything.\n\tif [
\"$(id -u)\" -eq 0 ]; then\n\t\techo \"Running as root, skipping setcap\"\n\t\t#
Warn the user if root does not have CAP_NET_ADMIN.\n\t\tif ! capsh --has-p=CAP_NET_ADMIN;
then\n\t\t\tprintnetadminmissing\n\t\tfi\n\n\t# If not running as root,
make sure we have sudo perms and the \"setcap\" +\n\t# \"capsh\" binaries
exist.\n\telif sudo -nl && haslibcap2; then\n\t\t# Make sure the root user
has CAP_NET_ADMIN.\n\t\tif sudo -n capsh --has-p=CAP_NET_ADMIN; then\n\t\t\tsudo
-n setcap CAP_NET_ADMIN=+ep ./$BINARY_NAME || true\n\t\telse\n\t\t\tprintnetadminmissing\n\t\tfi\n\n\t#
If we are not running as root, cant sudo, and \"setcap\" does not exist,
we\n\t# cannot do anything.\n\telse\n\t\techo \"Unable to setcap agent binary.
To enable improved network performance, \" + \\\n\t\t\t\"give the agent
passwordless sudo permissions and the \\\"setcap\\\" + \\\"capsh\\\" binaries.\"\n\t\techo
\"This has security implications. See https://man7.org/linux/man-pages/man7/capabilities.7.html\"\n\tfi\nfi\n\nexport
CODER_AGENT_AUTH=\"token\"\nexport CODER_AGENT_URL=\"https://coder.company.com/\"\nexec
./$BINARY_NAME agent\n\n EOT\n\n exec /sbin/init\n"
env:
- name: CODER_AGENT_TOKEN
value: XXXXX
- name: SYSBOX_ALLOW_TRUSTED_XATTR
value: "FALSE"
image: us.gcr.io/XXX/docker-image-systemd
imagePullPolicy: IfNotPresent
name: coder-MYUSER-0
resources:
limits:
cpu: "1"
memory: 4Gi
requests:
cpu: "1"
memory: 4Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /ws
mountPropagation: None
name: data
subPath: workspaces
- mountPath: /home
mountPropagation: None
name: data
subPath: home
- mountPath: /var/lib/docker
mountPropagation: None
name: data
subPath: var/lib/docker
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: coder-MYUSER-0
restartPolicy: Always
runtimeClassName: sysbox-runc
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
runAsNonRoot: false
runAsUser: 0
shareProcessNamespace: false
terminationGracePeriodSeconds: 30
volumes:
- name: data
persistentVolumeClaim:
claimName: coder-e832bafe-2d57-4d56-8e53-a807a86d0869
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the embedded pod.yaml, focusing on the PVC subPath mount at /var/lib/docker, fsGroup settings, and the sysbox-runc runtime. Reproduce the ownership change across pod restarts or node rescheduling and compare the mount state; done means identifying the cause and a safe way to preserve the PVC data and expected ownership.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, gcp, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100