hashicorp / hashicorp/consul

Consul image can not be started on Kubernetes/Openshift without mounted volume

Open
#21,762 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
When official Consul docker image is started on Kubernetes without mounted volume, it fails with either su-exec: setgroups(1000): Operation not permitted or failed to write NodeID to disk error.
---

#### Reproduction Steps

Steps for Openshift, steps for K8s should be similar:

1. Login into OpenShift
2. Create new project: `oc new-project ts-consul`
3. Create file consul.yml with following content:
```
---
apiVersion: "v1"
kind: "List"
items:
- apiVersion: "v1"
kind: "Service"
metadata:
name: "consul"
spec:
ports:
- name: "http"
port: 8500
targetPort: 8500
selector:
deployment: "consul"
type: "ClusterIP"
- apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "consul"
spec:
replicas: 1
selector:
matchLabels:
deployment: "consul"
template:
metadata:
labels:
deployment: "consul"
spec:
containers:
- image: "docker.io/hashicorp/consul:1.19"
# env:
# - name: "CONSUL_DISABLE_PERM_MGMT"
# value: "yes"
imagePullPolicy: "IfNotPresent"
name: "consul"
ports:
- containerPort: 8500
name: "http"
protocol: "TCP"

```
3. Deploy the container: `oc apply -f consul.yml -n ts-consul`
4. Start the container: `oc scale deployemnt/consul --replicas=1 -n ts-consul`
5. Wait for several seconds and check status:
```
$ oc get pods
NAME READY STATUS RESTARTS AGE
consul-6b486f7bfc-kjcd4 0/1 CrashLoopBackOff 3 (15s ago) 56s
```
6. Check pod logs: `oc logs pod/consul-6b486f7bfc-kjcd4` (replace with the id of your pod): `su-exec: setgroups(1000): Operation not permitted

#### Alternative solution

We can follow the solution, implemented in https://github.com/hashicorp/docker-consul/issues/103 and add `CONSUL_DISABLE_PERM_MGMT` property. Unfortunately, this will just lead to a different error:
```
failed to setup node ID: failed to write NodeID to disk: open /consul/data/node-id: permission denied
```

### Consul info for both Client and Server
We use official docker container `docker.io/hashicorp/consul:1.19`

### Operating system and Environment details

OC: ` 6.10.8-200.fc40.x86_64`
Openshift version:
```
Client Version: 4.16.10
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Kubernetes Version: v1.29.7+4510e9c
```

Similar error was previously described several times:
1. https://github.com/hashicorp/consul/issues/4172 — suggested solution is to use custom Docker image
2. https://github.com/hashicorp/docker-consul/issues/103 — added `CONSUL_DISABLE_PERM_MGMT` environment property, not helpful in this case, see "Alternative solution" section)
3. https://github.com/hashicorp/consul/issues/10403 —(recommended solution is to check "mount parameters", but that requires volume mounting, which would be overkill in some cases(e.g/ training or integration testing).

Usage of bitnami/consul image can be considered a workaround, but it comes with its own challenges[1] so it is preferable to have this issue solved for the official image.

Was earlier reported in this repo (https://github.com/hashicorp/consul/issues/12882) and in the docker-consul one (https://github.com/hashicorp/docker-consul/issues/184)

[1] https://github.com/bitnami-labs/sealed-secrets/issues/822

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the consul.yml manifest and the documented OpenShift commands, then inspect the pod logs for the su-exec and node-id permission errors. The issue names no repository files or tests; done means the official Consul image starts on Kubernetes or OpenShift without a mounted volume and without either permission failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.