canonical / canonical/mongo-single-kernel-library
MongoDB fails to restart pbm after client TLS is enabled
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 22
Description
## Steps to reproduce
1. juju deploy mongodb-k8s --channel=8/edge -n 5
2. juju deploy self-signed-certificates
3. juju integrate mongodb-k8s:peer-certificates self-signed-certificates
4. Wait active/idle after all the units restart
5. juju integrate mongodb-k8s:client-certificates self-signed-certificates
6. All the units go into blocked state
7. After a while the error resolves it self
## Expected behavior
The mongodb units never go into this blocked state. client tls enable is smooth.
## Actual behavior
The units goes into blocked status saying restart fail and that they will retry.
## Versions
Operating system: 24.04
Juju CLI: 3.6.23
Juju agent:
Charm revision: 203
LXD:
## Log output
Juju debug log:
```
unit-mongodb-k8s-1: 18:17:07 INFO unit.mongodb-k8s/1.juju-log rollingops-peers:0: single_kernel_mongo.managers.config:Restarting the PBM agent.
unit-mongodb-k8s-1: 18:17:12 ERROR unit.mongodb-k8s/1.juju-log rollingops-peers:0: single_kernel_mongo.core.k8s_workload:Change Error: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
----- Logs from task 0 -----
2026-06-05T16:17:12Z INFO Service "pbm-agent" already stopped.
----- Logs from task 1 -----
2026-06-05T16:17:12Z INFO Most recent service output:
(...)
2026-06-05T16:16:01.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:06.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:11.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:16.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:21.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:26.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:31.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:39.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:39.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:40.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:47.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:48.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:49.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:55.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:56.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:57.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:04.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:04.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:05.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:12.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:12Z ERROR service start attempt: exited quickly with code 1, will restart
-----
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/single_kernel_mongo/core/k8s_workload.py", line 70, in restart
self.container.restart(self.service)
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 2613, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2401, in restart_services
return self._services_action('restart', services, timeout, delay)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2428, in _services_action
raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
----- Logs from task 0 -----
2026-06-05T16:17:12Z INFO Service "pbm-agent" already stopped.
----- Logs from task 1 -----
2026-06-05T16:17:12Z INFO Most recent service output:
(...)
2026-06-05T16:16:01.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:06.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:11.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:16.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:21.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:26.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:31.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:39.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:39.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:40.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:47.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:48.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:49.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:55.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:56.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:57.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:04.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:04.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:05.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:12.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:12Z ERROR service start attempt: exited quickly with code 1, will restart
-----
unit-mongodb-k8s-1: 18:17:12 ERROR unit.mongodb-k8s/1.juju-log rollingops-peers:0: single_kernel_mongo.managers.config:Failed to restart pbm-agent: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
unit-mongodb-k8s-1: 18:17:12 INFO unit.mongodb-k8s/1.juju-log rollingops-peers:0: single_kernel_mongo.events.lifecycle:Failed to restart services: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/single_kernel_mongo/core/k8s_workload.py", line 70, in restart
self.container.restart(self.service)
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 2613, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2401, in restart_services
return self._services_action('restart', services, timeout, delay)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2428, in _services_action
raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
----- Logs from task 0 -----
2026-06-05T16:17:12Z INFO Service "pbm-agent" already stopped.
----- Logs from task 1 -----
2026-06-05T16:17:12Z INFO Most recent service output:
```
## Additional context
Contributor guide
Research direction
Start with the restart path in single_kernel_mongo/core/k8s_workload.py and the pbm-agent logs showing the missing /etc/mongod/external-ca.crt file. Reproduce the sequence with mongodb-k8s, self-signed-certificates, and the two integrations, then trace how client TLS configuration reaches PBM. Done means enabling client TLS does not leave units blocked and pbm-agent restarts successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, python
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100