Graylog2 / Graylog2/graylog-helm

MongoDB Version 7.0 silently fails

Open
#89 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go Template
Stars
12
Forks
3
Avg merge
2d 4h
Merged PRs (30d)
13

Description

When I supply `--set mongodb.version="7.0"` the mongo pods achieve a Ready state but the underlying application is clearly not healthy and Graylog and Data Node pods timeout when connecting to them.

However, specifying version `7.0.32` (7.0.latest) works fine.

Here are the mongo pod logs when using `7.0`:
```
│ mongodb-agent [2026-05-07T20:02:34.308+0000] [.debug] [util/distros/distros.go:LinuxFlavorAndVersionUncached:140] Detected linux flavor rhel version 9.6 │
│ mongod 2026-05-07T20:02:34.057Z INFO versionhook/main.go:33 Running version change post-start hook │
│ mongod 2026-05-07T20:02:34.058Z INFO versionhook/main.go:40 Waiting for agent health status... │
│ mongod 2026-05-07T20:02:35.058Z INFO versionhook/main.go:46 Agent health status file not found, mongod will start │
│ mongod Waiting for config and keyfile files to be created by the agent... │
│ mongod Waiting... │
│ mongod Waiting... │
│ mongod Waiting... │
│ mongod Waiting... │
│ mongod Waiting...
```

Here's the `describe`:

```
Name: graylog-mongo-rs-0
Namespace: graylog
Priority: 0
Service Account: graylog-mongo-sa
Node: ip-172-31-21-46.ec2.internal/172.31.21.46
Start Time: Thu, 07 May 2026 16:02:18 -0400
Labels: app=graylog-mongo-rs-svc
apps.kubernetes.io/pod-index=0
controller-revision-hash=graylog-mongo-rs-64f869fb7f
statefulset.kubernetes.io/pod-name=graylog-mongo-rs-0
topology.kubernetes.io/region=us-east-1
topology.kubernetes.io/zone=us-east-1a
Annotations: agent.mongodb.com/version: 1
Status: Running
IP: 172.31.22.214
IPs:
IP: 172.31.22.214
Controlled By: StatefulSet/graylog-mongo-rs
Init Containers:
mongod-posthook:
Container ID: containerd://f4ce55dece34e5e60027c63cac5bc6aef5dfbe98f2f127183b4b3f2538d155a3
Image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.10
Image ID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:f321ec1d25d6e98805b8be9321f2a926d702835136dde88d5fffe917c2df1d0a
Port:
Host Port:
Command:
cp
version-upgrade-hook
/hooks/version-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 07 May 2026 16:02:32 -0400
Finished: Thu, 07 May 2026 16:02:32 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500M
Requests:
cpu: 500m
memory: 400M
Environment:
Mounts:
/hooks from hooks (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rspwn (ro)
mongodb-agent-readinessprobe:
Container ID: containerd://fbda4639ae2b64313313cbec4020ff6eac6e1c5b6cf8a697199b42543cfec18b
Image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.23
Image ID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:436fc328f3887f022a4760afd03da1a7091d285baf3d627a17d80bbdaab0ee47
Port:
Host Port:
Command:
cp
/probes/readinessprobe
/opt/scripts/readinessprobe
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 07 May 2026 16:02:33 -0400
Finished: Thu, 07 May 2026 16:02:33 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500M
Requests:
cpu: 500m
memory: 400M
Environment:
Mounts:
/opt/scripts from agent-scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rspwn (ro)
Containers:
mongod:
Container ID: containerd://bb7efd65c7d83d73de93f101e48a726e0f80873f770d48d1db2483a818572bc2
Image: quay.io/mongodb/mongodb-community-server:7.0-ubi8
Image ID: quay.io/mongodb/mongodb-community-server@sha256:48c57f0883d4b8f358aabe5f02a03dd1205d178c37c861e39267eda422d100e8
Port:
Host Port:
Command:
/bin/sh
-c

if [ -e "/hooks/version-upgrade" ]; then
#run post-start hook to handle version changes (if exists)
/hooks/version-upgrade
fi

# wait for config and keyfile to be created by the agent
echo "Waiting for config and keyfile files to be created by the agent..."
while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do
sleep 3;
echo "Waiting..."
done

# sleep is important after agent issues shutdown command
# k8s restarts the mongod container too quickly for the agent to realize mongod is down
echo "Sleeping for 15s..."
sleep 15

# start mongod with this configuration
echo "Starting mongod..."
exec mongod -f /data/automation-mongod.conf

Args:

State: Running
Started: Thu, 07 May 2026 16:02:34 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500M
Requests:
cpu: 500m
memory: 400M
Environment:
AGENT_STATUS_FILEPATH: /healthstatus/agent-health-status.json
Mounts:
/data from data-volume (rw)
/healthstatus from healthstatus (rw)
/hooks from hooks (rw)
/tmp from tmp (rw)
/var/lib/mongodb-mms-automation/authentication from graylog-mongo-rs-keyfile (rw)
/var/log/mongodb-mms-automation from logs-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rspwn (ro)
mongodb-agent:
Container ID: containerd://cdce1bbdc9c9cb8279f451c2eacf800b89f2afd6e1f6960eda831fb7ddac0bba
Image: quay.io/mongodb/mongodb-agent:108.0.2.8729-1
Image ID: quay.io/mongodb/mongodb-agent@sha256:b422829aac63f068aed4a6948fd14cbd3c3bfad6d47ab76523f6839bf20b0203
Port:
Host Port:
Command:
/bin/bash
-c
current_uid=$(id -u)
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
State: Running
Started: Thu, 07 May 2026 16:02:34 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500M
Requests:
cpu: 500m
memory: 400M
Readiness: exec [/opt/scripts/readinessprobe] delay=5s timeout=1s period=10s #success=1 #failure=40
Environment:
AGENT_STATUS_FILEPATH: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
AUTOMATION_CONFIG_MAP: graylog-mongo-rs-config
HEADLESS_AGENT: true
POD_NAMESPACE: graylog (v1:metadata.namespace)
Mounts:
/data from data-volume (rw)
/opt/scripts from agent-scripts (rw)
/tmp from tmp (rw)
/var/lib/automation/config from automation-config (ro)
/var/lib/mongodb-mms-automation/authentication from graylog-mongo-rs-keyfile (rw)
/var/log/mongodb-mms-automation from logs-volume (rw)
/var/log/mongodb-mms-automation/healthstatus from healthstatus (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rspwn (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-volume-graylog-mongo-rs-0
ReadOnly: false
logs-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: logs-volume-graylog-mongo-rs-0
ReadOnly: false
agent-scripts:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
automation-config:
Type: Secret (a volume populated by a Secret)
SecretName: graylog-mongo-rs-config
Optional: false
graylog-mongo-rs-keyfile:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
healthstatus:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
hooks:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-rspwn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m12s default-scheduler Successfully assigned graylog/graylog-mongo-rs-0 to ip-172-31-21-46.ec2.internal
Normal SuccessfulAttachVolume 3m1s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-3841c815-8c23-461f-9932-b805a66dcdfc"
Normal SuccessfulAttachVolume 2m59s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-13c2f262-6a93-4116-b7f5-dd8395cc7c1d"
Normal Pulling 2m58s kubelet Pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.10"
Normal Pulled 2m58s kubelet Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.10" in 78ms (78ms including waiting). Image size: 59082744 bytes.
Normal Created 2m58s kubelet Container created
Normal Started 2m58s kubelet Container started
Normal Pulling 2m58s kubelet Pulling image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.23"
Normal Pulled 2m58s kubelet Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.23" in 70ms (70ms including waiting). Image size: 60690112 bytes.
Normal Created 2m58s kubelet Container created
Normal Started 2m57s kubelet Container started
Normal Pulled 2m57s kubelet Container image "quay.io/mongodb/mongodb-community-server:7.0-ubi8" already present on machine and can be accessed by the pod
Normal Created 2m57s kubelet Container created
Normal Started 2m56s kubelet Container started
Normal Pulling 2m56s kubelet Pulling image "quay.io/mongodb/mongodb-agent:108.0.2.8729-1"
Normal Pulled 2m56s kubelet Successfully pulled image "quay.io/mongodb/mongodb-agent:108.0.2.8729-1" in 81ms (81ms including waiting). Image size: 182416151 bytes.
Normal Created 2m56s kubelet Container created
Normal Started 2m56s kubelet Container started
```

Looks like `7.0` is a valid version value, and I can confirm `7.0.32` is running in the container:

```
bash-4.4$ mongod --version
db version v7.0.32
Build Info: {
"version": "7.0.32",
"gitVersion": "40a2e0f542b0a68f0ed51fa216e9d38b9917fe14",
"openSSLVersion": "OpenSSL 1.1.1k FIPS 25 Mar 2021",
"modules": [],
"allocator": "tcmalloc",
"environment": {
"distmod": "rhel80",
"distarch": "x86_64",
"target_arch": "x86_64"
}
}
```

But I can't connect to the mongo shell like I can when installing with `7.0.32`:

```
bash-4.4$ mongosh
Warning: Could not access file: EACCES: permission denied, mkdir '/data/db/.mongodb'
Current Mongosh Log ID: 69fcf29d8d38bb04b98ce5af
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.6.0
MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.