cockroachdb / cockroachdb/cockroach-operator
Permission denied creating the data directory
- Dominant language
- Go
- Stars
- 318
- Forks
- 104
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
I'm stuck with the following error when trying to create any kind of CockroachDB cluster using the operator:
```plain
E240215 20:18:40.885312 1 1@cli/clierror/check.go:35 [-] 1 ERROR: connection lost.
E240215 20:18:40.885312 1 1@cli/clierror/check.go:35 [-] 1 +creating data directory: mkdir /cockroach/cockroach-data/auxiliary: permission denied
ERROR: connection lost.
creating data directory: mkdir /cockroach/cockroach-data/auxiliary: permission denied
Failed running "start"
```
The cluster manifest might look like this:
```yaml
apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
name: primary-crdb
spec:
cockroachDBVersion: v23.1.11
dataStore:
pvc:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "1Gi"
storageClassName: primary-nfs
volumeMode: Filesystem
nodes: 3
resources:
limits:
cpu: 2
memory: 2Gi
requests:
cpu: 500m
memory: 2Gi
tlsEnabled: true
```
The storage class is for `csi-driver-nfs` and leads to the following directory tree:
```plain
$ ls -lahF //*
//pvc-40b518b5-bccc-4610-b804-0bd2175f5eed:
total 18K
drwxrwsr-x 2 root 1000581000 2 Feb 11 16:26 ./
drwxr-xr-x 6 root root 6 Feb 11 17:07 ../
```
The CockroachDB pod manifest (`kubectl get pods primary-crdb-0 --output yaml`) has the following security context:
```yaml
securityContext:
fsGroup: 1000581000
runAsUser: 1000581000
```
Which explains why the permissions actually don't add up.
For comparison, using this storage setup, it is possible to create a working mount like this:
```yaml
...
containers:
- name: busybox
image: busybox:1.28
command: [ "sh", "-c", "sleep 1h" ]
volumeMounts:
- name: data
mountPath: "/test"
securityContext:
runAsUser: 2000
runAsGroup: 2000
fsGroup: 2000
volumes:
- name: data
persistentVolumeClaim:
claimName: test
```
When creating a file (`touch /test/file`) from inside the container the directory tree looks like this:
```plain
$ ls -lahF //*
//pvc-730e175e-af46-4e48-b4e4-5a1dd568307d:
total 19K
drwxrwsr-x 2 root 2000 3 Feb 11 17:14 ./
drwxr-xr-x 6 root root 6 Feb 11 17:07 ../
-rw-rw-r-- 1 2000 2000 0 Feb 11 17:14 file
```
It works because all owner and group match.
I'm wondering if the operator should specify `runAsGroup` or if there is something unusual with my setup, and if this should not be necessary at all.
The locations in the code would be the following:
- https://github.com/cockroachdb/cockroach-operator/blob/v2.12.0/pkg/resource/statefulset.go#L208
- https://github.com/cockroachdb/cockroach-operator/blob/v2.12.0/pkg/resource/job.go#L95
Even though I don't have much experience in self-hosting storage for Kubernetes, I would say adding `runAsGroup` is the right idea and I'm happy to create a PR if wanted.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.