nextcloud / nextcloud/helm

nextcloud-nginx container crashlooping after securityContext update; `/var/www/html/config` always owned by root

Open
#335 21 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Persistence securityContext
Dominant language
Go Template
Stars
534
Forks
314
Avg merge
42m
Merged PRs (30d)
1

Description

Description

I've edited this for full context of how we got here, as this issue is getting kind of long, because it needed to be tested in a lot of different ways which lead me in several directions.

This issue is a continuation of the conversation started after #269 was merged. I was originally trying to changed the podSecurityContext.runAsUser and podSecurityContext.runAsGroup to 33 because I was trying to diagnose why the /var/www/html/config directory was always owned by root. I am deploying the nextcloud helm chart using persistent volumes on k3s with the default local path provisioner.

I learned that the podSecurityContext.fsGroup was always being set to 82 anytime you used nginx.enabled and didn't set podSecurityContext.fsGroup explicitly, so I submitted a draft PR here to fix it to so that it checks image.flavor for alpine: https://github.com/nextcloud/helm/pull/379

Through the comments here you can see other things I'm currently testing, because I'm still not sure is it's just the local path provisioner on k3s or k3s itself or what, but the best I can get is 🤷 I'll update this issue description with more clarity as it comes.

Original Issue that was opened on Jan 23

The nginx container in the nextcloud pod won't start and complains of a readonly file system, even if I try to only set the nextcloud.securityContext.

I have created a new cluster and deployed nextcloud with the securityContext parameters from the values.yaml of this repo, including the nginx security context. My entire values.yaml is here, but the parts that matter are:

securityContext parameters in my old `values.yaml` 
nextcloud:
  # securityContext parameters. For example you may need to define runAsNonRoot directive
  securityContext:
    runAsUser: 33
    runAsGroup: 33
    runAsNonRoot: true
    readOnlyRootFilesystem: false

  # securityContext parameters. For example you may need to define runAsNonRoot directive
  podSecurityContext:
    runAsUser: 33
    runAsGroup: 33
    runAsNonRoot: true
    readOnlyRootFilesystem: false

...

  nginx:
    ## You need to set an fpm version of the image for nextcloud if you want to use nginx!
    enabled: true
    image:
      repository: nginx
      tag: alpine
      pullPolicy: Always
    
    # this is copied almost directly from the values.yaml, but I changed readOnlyRootFilesystem to false while testing
    securityContext:
      runAsUser: 82
      runAsGroup: 33
      runAsNonRoot: true
      readOnlyRootFilesystem: false

The nextcloud pod is in a crashloopbackoff state with the offending container being nginx, and this being the logs:

│ 2023-01-23T15:44:59.428798413+01:00 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration        │
│ 2023-01-23T15:44:59.428820874+01:00 /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/                               │
│ 2023-01-23T15:44:59.429173908+01:00 /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh                   │
│ 2023-01-23T15:44:59.429979412+01:00 10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file sy │
│ stem?)                                                                                                                                      │
│ 2023-01-23T15:44:59.430071429+01:00 /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh                       │
│ 2023-01-23T15:44:59.431167356+01:00 /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh                       │
│ 2023-01-23T15:44:59.431715519+01:00 /docker-entrypoint.sh: Configuration complete; ready for start up                                       │
│ 2023-01-23T15:44:59.433513935+01:00 2023/01/23 14:44:59 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)  │
│ 2023-01-23T15:44:59.433519229+01:00 nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)                    │
│ 2023-01-23T14:45:24.296336176Z Stream closed EOF for nextcloud/nextcloud-web-app-66fc5dfcb7-kxlnp (nextcloud-nginx)

This is the resulting deployment.yaml when I do a kubectl get deployment -n nextcloud nextcloud-web-app > deployment.yaml:

Click me for the nextcloud deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2023-01-23T14:43:58Z"
  generation: 52
  labels:
    app.kubernetes.io/component: app
    app.kubernetes.io/instance: nextcloud-web-app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nextcloud
    argocd.argoproj.io/instance: nextcloud-web-app
    helm.sh/chart: nextcloud-3.4.1
  name: nextcloud-web-app
  namespace: nextcloud
  resourceVersion: "3340"
  uid: cde1dd07-103a-4c04-931d-071ab3c5b448
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: app
      app.kubernetes.io/instance: nextcloud-web-app
      app.kubernetes.io/name: nextcloud
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        nextcloud-config-hash: d1d9ac6f86f643b460f8e4e8e886b65382ad49aede8762f8ea74ccd86b7e3f28
        nginx-config-hash: 16c61772d9e74de7322870fd3a045598ea01f6e16be155d116423e6a246dcddc
        php-config-hash: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: app
        app.kubernetes.io/instance: nextcloud-web-app
        app.kubernetes.io/name: nextcloud
    spec:
      containers:
      - env:
        - name: POSTGRES_HOST
          value: nextcloud-web-app-postgresql
        - name: POSTGRES_DB
          value: nextcloud
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: nextcloud-pgsql-credentials
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: nextcloudPassword
              name: nextcloud-pgsql-credentials
        - name: NEXTCLOUD_ADMIN_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: nextcloud-admin-credentials
        - name: NEXTCLOUD_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: nextcloud-admin-credentials
        - name: NEXTCLOUD_TRUSTED_DOMAINS
          value: nextcloud.vleermuis.tech
        - name: NEXTCLOUD_DATA_DIR
          value: /var/www/html/data
        image: nextcloud:25.0.3-fpm
        imagePullPolicy: Always
        name: nextcloud
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/www/
          name: nextcloud-main
          subPath: root
        - mountPath: /var/www/html
          name: nextcloud-main
          subPath: html
        - mountPath: /var/www/html/data
          name: nextcloud-main
          subPath: data
        - mountPath: /var/www/html/config
          name: nextcloud-main
          subPath: config
        - mountPath: /var/www/html/custom_apps
          name: nextcloud-main
          subPath: custom_apps
        - mountPath: /var/www/tmp
          name: nextcloud-main
          subPath: tmp
        - mountPath: /var/www/html/themes
          name: nextcloud-main
          subPath: themes
        - mountPath: /var/www/html/config/logging.config.php
          name: nextcloud-config
          subPath: logging.config.php
        - mountPath: /var/www/html/config/proxy.config.php
          name: nextcloud-config
          subPath: proxy.config.php
        - mountPath: /var/www/html/config/.htaccess
          name: nextcloud-config
          subPath: .htaccess
        - mountPath: /var/www/html/config/apache-pretty-urls.config.php
          name: nextcloud-config
          subPath: apache-pretty-urls.config.php
        - mountPath: /var/www/html/config/apcu.config.php
          name: nextcloud-config
          subPath: apcu.config.php
        - mountPath: /var/www/html/config/apps.config.php
          name: nextcloud-config
          subPath: apps.config.php
        - mountPath: /var/www/html/config/autoconfig.php
          name: nextcloud-config
          subPath: autoconfig.php
        - mountPath: /var/www/html/config/redis.config.php
          name: nextcloud-config
          subPath: redis.config.php
        - mountPath: /var/www/html/config/smtp.config.php
          name: nextcloud-config
          subPath: smtp.config.php
      - image: nginx:alpine
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Host
              value: nextcloud.vleermuis.tech
            path: /status.php
            port: http
            scheme: HTTP
          initialDelaySeconds: 45
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 5
        name: nextcloud-nginx
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Host
              value: nextcloud.vleermuis.tech
            path: /status.php
            port: http
            scheme: HTTP
          initialDelaySeconds: 45
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        securityContext:
          readOnlyRootFilesystem: false
          runAsGroup: 33
          runAsNonRoot: true
          runAsUser: 82
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/www/
          name: nextcloud-main
          subPath: root
        - mountPath: /var/www/html
          name: nextcloud-main
          subPath: html
        - mountPath: /var/www/html/data
          name: nextcloud-main
          subPath: data
        - mountPath: /var/www/html/config
          name: nextcloud-main
          subPath: config
        - mountPath: /var/www/html/custom_apps
          name: nextcloud-main
          subPath: custom_apps
        - mountPath: /var/www/tmp
          name: nextcloud-main
          subPath: tmp
        - mountPath: /var/www/html/themes
          name: nextcloud-main
          subPath: themes
        - mountPath: /etc/nginx/nginx.conf
          name: nextcloud-nginx-config
          subPath: nginx.conf
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - until pg_isready -h nextcloud-web-app-postgresql -U ${POSTGRES_USER} ; do
          sleep 2 ; done
        env:
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: nextcloud-pgsql-credentials
        image: bitnami/postgresql:14.4.0-debian-11-r23
        imagePullPolicy: IfNotPresent
        name: postgresql-isready
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 82
        runAsGroup: 33
        runAsNonRoot: true
        runAsUser: 33
      serviceAccount: nextcloud-serviceaccount
      serviceAccountName: nextcloud-serviceaccount
      terminationGracePeriodSeconds: 30
      volumes:
      - name: nextcloud-main
        persistentVolumeClaim:
          claimName: nextcloud-files
      - configMap:
          defaultMode: 420
          name: nextcloud-web-app-config
        name: nextcloud-config
      - configMap:
          defaultMode: 420
          name: nextcloud-web-app-nginxconfig
        name: nextcloud-nginx-config
status:
  conditions:
  - lastTransitionTime: "2023-01-23T14:43:58Z"
    lastUpdateTime: "2023-01-23T14:43:58Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2023-01-23T14:43:58Z"
    lastUpdateTime: "2023-01-23T14:43:58Z"
    message: ReplicaSet "nextcloud-web-app-66fc5dfcb7" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 52
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

Where does the UID 82 come from? (Edit: it comes from the alpine nextcloud and nginx images - that's www-data)

I set that to 33 (nextcloud's www-data user) to test, but it didn't seem to make a difference. Just so it's clear, without editing any of the security contexts, everything works, but I would like the security context to work, because otherwise it causes my restores from backups to fail, because the /var/www/html/config directory is always created with root ownership, which means if the restores run as www-data, they can't restore that particular directory, which is important. I'm hoping the security context fixes that, so that nothing has to run as root in this stack.

I'm deploying the 3.4.1 nextcloud helm chart via Argo CD onto k3s on Ubuntu 22.04.
Update: problem still present in 3.5.7 helm chart.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the securityContext settings in values.yaml and compare them with the rendered deployment.yaml shown in the report, including the nginx container and pod fsGroup. Review the draft PR #379 and the issue's testing history to identify the intended behavior; done means the nginx container starts and the config directory ownership is correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.