cockroachdb / cockroachdb/cockroach-operator

Permission denied creating the data directory

Open
#1,016 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
318
Forks
104
Avg merge
1d 6h
Merged PRs (30d)
1

Description

I'm stuck with the following error when trying to create any kind of CockroachDB cluster using the operator:

```plain
E240215 20:18:40.885312 1 1@cli/clierror/check.go:35 [-] 1 ERROR: connection lost.
E240215 20:18:40.885312 1 1@cli/clierror/check.go:35 [-] 1 +creating data directory: mkdir /cockroach/cockroach-data/auxiliary: permission denied
ERROR: connection lost.

creating data directory: mkdir /cockroach/cockroach-data/auxiliary: permission denied
Failed running "start"
```

The cluster manifest might look like this:

```yaml
apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
name: primary-crdb
spec:
cockroachDBVersion: v23.1.11
dataStore:
pvc:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "1Gi"
storageClassName: primary-nfs
volumeMode: Filesystem
nodes: 3
resources:
limits:
cpu: 2
memory: 2Gi
requests:
cpu: 500m
memory: 2Gi
tlsEnabled: true
```

The storage class is for `csi-driver-nfs` and leads to the following directory tree:

```plain
$ ls -lahF //*
//pvc-40b518b5-bccc-4610-b804-0bd2175f5eed:
total 18K
drwxrwsr-x 2 root 1000581000 2 Feb 11 16:26 ./
drwxr-xr-x 6 root root 6 Feb 11 17:07 ../
```

The CockroachDB pod manifest (`kubectl get pods primary-crdb-0 --output yaml`) has the following security context:

```yaml
securityContext:
fsGroup: 1000581000
runAsUser: 1000581000
```

Which explains why the permissions actually don't add up.

For comparison, using this storage setup, it is possible to create a working mount like this:

```yaml
...
containers:
- name: busybox
image: busybox:1.28
command: [ "sh", "-c", "sleep 1h" ]
volumeMounts:
- name: data
mountPath: "/test"
securityContext:
runAsUser: 2000
runAsGroup: 2000
fsGroup: 2000
volumes:
- name: data
persistentVolumeClaim:
claimName: test
```

When creating a file (`touch /test/file`) from inside the container the directory tree looks like this:

```plain
$ ls -lahF //*
//pvc-730e175e-af46-4e48-b4e4-5a1dd568307d:
total 19K
drwxrwsr-x 2 root 2000 3 Feb 11 17:14 ./
drwxr-xr-x 6 root root 6 Feb 11 17:07 ../
-rw-rw-r-- 1 2000 2000 0 Feb 11 17:14 file
```

It works because all owner and group match.

I'm wondering if the operator should specify `runAsGroup` or if there is something unusual with my setup, and if this should not be necessary at all.

The locations in the code would be the following:

- https://github.com/cockroachdb/cockroach-operator/blob/v2.12.0/pkg/resource/statefulset.go#L208
- https://github.com/cockroachdb/cockroach-operator/blob/v2.12.0/pkg/resource/job.go#L95

Even though I don't have much experience in self-hosting storage for Kubernetes, I would say adding `runAsGroup` is the right idea and I'm happy to create a PR if wanted.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.