canonical / canonical/mongo-single-kernel-library

the config-server status stays as "Draining shard" even if shards are not draining (TF - aws)

Open
#437 1 comment 0 reactions 1 assignee Claimed by @Gu1nness View on GitHub
bug
Dominant language
Python
Stars
1
Forks
2
Avg merge
2d 23h
Merged PRs (30d)
22

Description

When deploying a sharded cluster using the TF on aws, the config server reports "draining" shards even if shards are not being drained.

Image

This comes from this extract of code
```
with MongoConnection(self.state.mongos_config) as mongo:
cluster_shards = mongo.get_shard_members()

relation_shards = {relation.app.name for relation in self.state.config_server_relation}
if shard_draining := (cluster_shards - relation_shards):
draining = ",".join(shard_draining)
status = ConfigServerStatuses.draining_shard(draining)
charm_statuses["unit"].append(status)
charm_statuses["app"].append(status)
```

In a cross model relation, the `cluster_shards` and `relation_shards` reports different values
```
unit-config-server-0: 11:18:59 INFO unit.config-server/0.juju-log single_kernel_mongo.managers.sharding:cluster_shards {'shard-one', 'shard-two'}, relation_shards {'remote-72feaf65ebac4aff8e2455b52c92935e', 'remote-cd153d311b9e4ad58c71c443cd817d8a'}
```

## Steps to reproduce

1. Deploy a cross-model sharded cluster on aws using [TF](https://github.com/canonical/charms-reference-architectures/tree/main/solutions/data/charmed-mongodb/sharded-cluster/aws/vm)

## Expected behavior
The config-server is active/idle

## Actual behavior

```
config-server/0* maintenance idle 2 10.0.2.193 27017-27018/tcp Draining shard shard-two,shard-one...
```

## Versions

Operating system: 24.04

Juju CLI: 3.6.27

Juju agent:

Charm revision: mongodb 8/edge 374

LXD:

## Log output

Juju debug log:

## Additional context

The shards are active/idle

Image

Mongos reports the 2 shards:
```
$ sudo charmed-mongodb.mongosh --host localhost --port 27018 --tls --tlsCAFile /var/snap/charmed-mongodb/current/etc/mongod/external-ca.crt --tlsAllowInvalidHostnames --authenticationDatabase admin --username charmed-operator --password <>
Current Mongosh Log ID: 6a9aaa13c6ea3f7b5e3a02af
Connecting to: mongodb://@localhost:27018/?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCAFile=%2Fvar%2Fsnap%2Fcharmed-mongodb%2Fcurrent%2Fetc%2Fmongod%2Fexternal-ca.crt&tlsAllowInvalidHostnames=true&authSource=admin&appName=mongosh+2.9.0
Using MongoDB: 8.0.28-12
Using Mongosh: 2.9.0
mongosh 2.10.0 is available for download: https://www.mongodb.com/try/download/shell

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

------
The server generated these startup warnings when booting
2026-09-03T17:02:56.451+00:00: While invalid X509 certificates may be used to connect to this server, they will not be considered permissible for authentication
------

[direct: mongos] test> db.adminCommand({ listShards: 1 })
{
shards: [
{
_id: 'shard-one',
host: 'shard-one/10.0.2.166:27017,10.0.2.194:27017,10.0.2.64:27017',
state: 1,
topologyTime: Timestamp({ t: 1788514736, i: 10 }),
replSetConfigVersion: Long('-1')
},
{
_id: 'shard-two',
host: 'shard-two/10.0.2.104:27017,10.0.2.70:27017,10.0.2.80:27017',
state: 1,
topologyTime: Timestamp({ t: 1788515317, i: 10 }),
replSetConfigVersion: Long('-1')
}
],
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1788520985, i: 5 }),
signature: {
hash: Binary.createFromBase64('J3OHSjSyvgTzSculkpZP5d4uaPk=', 0),
keyId: Long('7681354726050365460')
}
},
operationTime: Timestamp({ t: 1788520985, i: 5 })
}
``

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.