canonical / canonical/mongo-single-kernel-library

MongoDB fails to restart pbm after client TLS is enabled

Open
#350 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1
Forks
2
Avg merge
2d 23h
Merged PRs (30d)
22

Description

## Steps to reproduce

1. juju deploy mongodb-k8s --channel=8/edge -n 5
2. juju deploy self-signed-certificates
3. juju integrate mongodb-k8s:peer-certificates self-signed-certificates
4. Wait active/idle after all the units restart
5. juju integrate mongodb-k8s:client-certificates self-signed-certificates
6. All the units go into blocked state
7. After a while the error resolves it self

## Expected behavior

The mongodb units never go into this blocked state. client tls enable is smooth.

## Actual behavior

The units goes into blocked status saying restart fail and that they will retry.

## Versions

Operating system: 24.04

Juju CLI: 3.6.23

Juju agent:

Charm revision: 203

LXD:

## Log output

Juju debug log:

```
unit-mongodb-k8s-1: 18:17:07 INFO unit.mongodb-k8s/1.juju-log rollingops-peers:0: single_kernel_mongo.managers.config:Restarting the PBM agent.
unit-mongodb-k8s-1: 18:17:12 ERROR unit.mongodb-k8s/1.juju-log rollingops-peers:0: single_kernel_mongo.core.k8s_workload:Change Error: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
----- Logs from task 0 -----
2026-06-05T16:17:12Z INFO Service "pbm-agent" already stopped.
----- Logs from task 1 -----
2026-06-05T16:17:12Z INFO Most recent service output:
(...)
2026-06-05T16:16:01.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:06.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:11.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:16.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:21.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:26.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:31.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:39.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:39.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:40.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:47.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:48.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:49.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:55.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:56.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:57.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:04.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:04.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:05.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:12.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:12Z ERROR service start attempt: exited quickly with code 1, will restart
-----
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/single_kernel_mongo/core/k8s_workload.py", line 70, in restart
self.container.restart(self.service)
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 2613, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2401, in restart_services
return self._services_action('restart', services, timeout, delay)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2428, in _services_action
raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
----- Logs from task 0 -----
2026-06-05T16:17:12Z INFO Service "pbm-agent" already stopped.
----- Logs from task 1 -----
2026-06-05T16:17:12Z INFO Most recent service output:
(...)
2026-06-05T16:16:01.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:06.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:11.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:16.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:21.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:26.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:31.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2026-06-05T16:16:39.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:39.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:40.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:47.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:48.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:49.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:55.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:56.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:16:57.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:04.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:04.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:05.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:12.000+0000 E Exit: connect to PBM: create mongo connection: invalid connection string option: open /etc/mongod/external-ca.crt: no such file or directory
2026-06-05T16:17:12Z ERROR service start attempt: exited quickly with code 1, will restart
-----
unit-mongodb-k8s-1: 18:17:12 ERROR unit.mongodb-k8s/1.juju-log rollingops-peers:0: single_kernel_mongo.managers.config:Failed to restart pbm-agent: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
unit-mongodb-k8s-1: 18:17:12 INFO unit.mongodb-k8s/1.juju-log rollingops-peers:0: single_kernel_mongo.events.lifecycle:Failed to restart services: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/single_kernel_mongo/core/k8s_workload.py", line 70, in restart
self.container.restart(self.service)
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 2613, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2401, in restart_services
return self._services_action('restart', services, timeout, delay)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-mongodb-k8s-1/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2428, in _services_action
raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "pbm-agent" (service start attempt: exited quickly with code 1, will restart)
----- Logs from task 0 -----
2026-06-05T16:17:12Z INFO Service "pbm-agent" already stopped.
----- Logs from task 1 -----
2026-06-05T16:17:12Z INFO Most recent service output:

```

## Additional context

Contributor guide

Open the contributing guide

Research direction

Start with the restart path in single_kernel_mongo/core/k8s_workload.py and the pbm-agent logs showing the missing /etc/mongod/external-ca.crt file. Reproduce the sequence with mongodb-k8s, self-signed-certificates, and the two integrations, then trace how client TLS configuration reaches PBM. Done means enabling client TLS does not leave units blocked and pbm-agent restarts successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.