Duplicate Key Error in MongoDB When Using Automatic Leader Election Mode

## **Expected Behavior**
- **Follower nodes should check the existing leader instead of attempting to write to the `cluster_locks` collection.**
- **The leader election mechanism should not cause unnecessary duplicate key errors.**
- **Each node should avoid unnecessary writes to MongoDB and only trigger the election process when necessary (e.g., when the leader changes or fails).**

---

## **Current Behavior**
- **Each node attempts to write its node information to the `cluster_locks` collection every time the configured interval elapses.**
- **This results in duplicate key errors in MongoDB, causing excessive log growth.**
- **When running a single node, no errors occur, but when two or more nodes are added, all nodes simultaneously generate duplicate key errors in MongoDB.**
- **No errors appear in the Graylog server logs, but MongoDB logs are constantly filled with duplicate key errors.**

---

## **Steps to Reproduce (for bugs)**
1. Configure **Graylog 6.1.7** with **MongoDB 7 and OpenSearch 2.15.0**.
2. Set `election_mode: automatic` and start the cluster.
3. Start a single **node** and check MongoDB logs (no errors occur).
4. Add a second **node** and check MongoDB logs (errors begin to appear).
5. **Each additional node generates duplicate key errors in MongoDB.**

---

## **Context**
This issue leads to **excessive growth in MongoDB logs** and **unnecessary processing overhead**.

- **Even though the Graylog cluster appears to function correctly, the constant duplicate key errors in MongoDB indicate a potential design issue.**
- **This can increase the load on MongoDB and negatively impact system performance.**
- **The excessive logging of these errors makes debugging and monitoring more difficult.**

---

## **Your Environment**

- **Graylog Version:** 6.1.7
- **OpenSearch Version:** 2.15.0
- **MongoDB Version:** 7
- **Operating System:** Ubuntu 24

---

## **MongoDB Error Logs**
```json
{
"t": { "$date": "2025-03-02T17:04:31.021+03:00" },
"s": "W",
"c": "WRITE",
"id": 7267501,
"ctx": "conn2627",
"msg": "Plan executor error during findAndModify",
"attr": {
"error": {
"code": 11000,
"codeName": "DuplicateKey",
"errmsg": "E11000 duplicate key error collection: graylogdb.cluster_locks index: resource_1 dup key: { resource: \"cluster-leader\" }",
"keyPattern": { "resource": 1 },
"keyValue": { "resource": "cluster-leader" }
},
"stats": {
"stage": "UPDATE",
"nReturned": 0,
"executionTimeMillisEstimate": 1,
"works": 2,
"advanced": 0,
"needTime": 1,
"needYield": 0,
"saveState": 0,
"restoreState": 0,
"failed": true,
"isEOF": 1,
"nMatched": 0,
"nWouldModify": 0,
"nWouldUpsert": 1,
"inputStage": {
"stage": "FETCH",
"filter": { "locked_by": { "$eq": "e3be38d1-d0a6-412b-be26-22b430c5ad6e" } },
"nReturned": 0,
"executionTimeMillisEstimate": 0,
"works": 2,
"advanced": 0,
"needTime": 1,
"needYield": 0,
"saveState": 0,
"restoreState": 0,
"isEOF": 1,
"docsExamined": 1,
"alreadyHasObj": 0,
"inputStage": {
"stage": "IXSCAN",
"nReturned": 1,
"executionTimeMillisEstimate": 0,
"works": 2,
"advanced": 1,
"needTime": 0,
"needYield": 0,
"saveState": 0,
"restoreState": 0,
"isEOF": 1,
"keyPattern": { "resource": 1 },
"indexName": "resource_1",
"isMultiKey": false,
"multiKeyPaths": { "resource": [] },
"isUnique": true,
"isSparse": false,
"isPartial": false,
"indexVersion": 2,
"direction": "forward",
"indexBounds": { "resource": ["[\"cluster-leader\", \"cluster-leader\"]"] },
"keysExamined": 1,
"seeks": 1,
"dupsTested": 0,
"dupsDropped": 0
}
}
}
}
}
```
---

## **Graylog configuration (server.conf)**
```
leader_election_mode = automatic
lock_service_lock_ttl = 60s
leader_election_lock_polling_interval = 5s
node_id_file = /etc/graylog/server/node-id
http_bind_address = 10.XX.XX.XX:9000
elasticsearch_hosts = http://admin:********@graylog-opensearch01.domain.local:9200
mongodb_uri = mongodb://admin:********@graylog-server01.domain.local:27017/graylogdb?authSource=admin&replicaSet=graylog
```