Secrets volumes not mounted with correct permissions

[Originally logged as a Minishift issue but after investigation there, identified as a core OpenShift Origin issue, hence @csrwng suggested to log the upstream issue here]

[tested on Minishift version v1.5.0+ae62cf2 - same behaviour for both default boot2docker and centos based VMs]

As part of my providing guidance around deploying MongoDB clusters to Kubernetes at http://k8smongodb.net/ I have built sample projects for various common k8s platforms. My latest addition looks at OpenShift (specifically trying it on Minishift), see the sample GitHub project for OpenShift at: (https://github.com/pkdone/openshift-mongodb-demo

To ensure the MongoDB cluster is secured by default, I create a Kubernetes secret first (using the 'kubectl create secret generic' command), before then deploying a StatefulSet, which declares the secrets volume to be mounted by each "mongod" container in the StatefulSet.

Example script I use to create the secret and deploy the StatefulSet: https://github.com/pkdone/openshift-mongodb-demo/blob/master/scripts/generate.sh

Example YAML resource file to define the StatefulSet (including secret volume mount configuration): https://github.com/pkdone/openshift-mongodb-demo/blob/master/resources/mongodb-service.yaml

However, when deployed, the "volumes - secret - defaultMode: 256" setting in the StatefulSet YAML resource is not being honoured by the Minishift environment. Once deployed, when I connect via the command line to each running"mongod" container, and I run:

$ ls -l /etc/secrets-volume/..data/internal-auth-mongodb-keyfile

In the terminal I see:

-r--r----- 1 root 1000000000 1004 Sep 4 10:12 /etc/secrets-volume/..data/internal-auth-mongodb-keyfile

However this file should not have any permissions set for group level, because "256" maps on to "0600" in Octal. Even when I change from "256" to some other value, although other parts of the permissions change accordingly for the file, the "read" permission is still always enabled for group.

This is an issue for using MongoDB with keyfiles as mongod won't even start if "group" or "other" have any permissions enabled on the keyfile (which I am storing as a secret). When the mongod process is started in the container, it errors with the following mongod specific error message:

2017-09-03T18:38:25.646+0000 I ACCESS [main] permissions on /etc/secrets-volume/internal-auth-mongodb-keyfile are too open

MongoDB's strictness around this is similar to other software out there, including how SSH sets permissions on local filesystems for SSH keys.

I don't get this issue when running the same scripts and resources on GKE, ACS and Minikube versions of Kubernetes (ie. there, mounted secret files correctly have 0600 permissions set).

This will cause an issue for anyone wanting to deploy a MongoDB cluster with authentication enabled (arguably the most common way to deploy MongoDB in Production).