In 3 node cluster (with q=1), when any one node fails and is re-added as blank, _security object is reset to default
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
## Your Environment
```
{
"couchdb": "Welcome",
"version": "3.1.1",
"git_sha": "ce596c65d",
"uuid": "8d406054df5edac06ee4906f3259e62f",
"features": [
"access-ready",
"partitioned",
"pluggable-storage-engines",
"reshard",
"scheduler"
],
"vendor": {
"name": "The Apache Software Foundation"
}
}
```
## Description
I have a 3 node couchdb `3.1.1` cluster with following configuration:
```
[cluster]
q=1
n=2
```
There is a non-partitioned database named `test2`, whose shards resides on `node1` and `node2`. `test2` database has following cluster settings:
```
"cluster":{"q":1,"n":2,"w":2,"r":2}
```
`test2` database has few documents and it's `_security` is:
```
{"admins":{"names":["superuser"],"roles":["admins"]},"members":{"names":["user1","user2"],"roles":["developers"]}}
```
I'm now running a scenario where any of the node's disk crashes. Let's say `node2`'s disk crashes.
I have performed following steps:
- Remove `node2` from the cluster
- Replace `node2` disk with a new blank disk
- Start `node2`
- Add `node2` into cluster
- It would resync the shards as its blank
- After a while, resync is completed
> At this stage, `test2` database's shards file(test2.1628258896.couch) can be seen on `node2`.
### 1: Now, when I retrieve `_security` for `test2` database from `node2`, it is reset to the default like:
```
{"members":{"roles":["_admin"]},"admins":{"roles":["_admin"]}}
```
If I retrieve `_security` from `node1` or `node3`, it responds with the correct: (which I set earlier before `node2` crash)
```
{"admins":{"names":["superuser"],"roles":["admins"]},"members":{"names":["user1","user2"],"roles":["developers"]}}
```
### 2: When I restart all nodes, there are following error logs shown:
```
node2 | [error] 2021-08-06T12:59:21.842335Z couchdb@node2 <0.4465.0> -------- Bad security object in <<"test2">>: [{{[{<<"members">>,{[{<<"roles">>,[<<"_admin">>]}]}},{<<"admins">>,{[{<<"roles">>,[<<"_admin">>]}]}}]},1},{{[{<<"admins">>,{[{<<"names">>,[<<"superuser">>]},{<<"roles">>,[<<"admins">>]}]}},{<<"members">>,{[{<<"names">>,[<<"user1">>,<<"user2">>]},{<<"roles">>,[<<"developers">>]}]}}]},1}]
```
### 3: Not relevant to crash, but relevant to same `_security` thing.
When 1 node is down, `PUT _security` fails with: `{"error":"error","reason":"no_majority"}`
- `node2` is down
- Create a new DB `test3`, whose shards would reside on `node1` and `node2`
- `PUT _security` for `test3`, it would fail with:
```
{"error":"error","reason":"no_majority"}
```
## Other observations
- `_sync_shards` throws same `Bad security object` error
- It works for `q=2`
- It fixes the issue when I updated `_security` of `test2` DB again
#### _sync_shards logs
```
node1 | [notice] 2021-08-06T13:38:55.064067Z couchdb@node1 <0.10379.1> c89bd8ac41 localhost:5984 172.20.0.1 admin POST /test2/_sync_shards 202 ok 2
node2 | [error] 2021-08-06T13:38:55.080064Z couchdb@node2 <0.7232.0> -------- Bad security object in <<"test2">>: [{{[{<<"members">>,{[{<<"roles">>,[<<"_admin">>]}]}},{<<"admins">>,{[{<<"roles">>,[<<"_admin">>]}]}}]},1},{{[{<<"admins">>,{[{<<"names">>,[<<"superuser">>]},{<<"roles">>,[<<"admins">>]}]}},{<<"members">>,{[{<<"names">>,[<<"user1">>,<<"user2">>]},{<<"roles">>,[<<"developers">>]}]}}]},1}]
node1 | [error] 2021-08-06T13:38:55.080503Z couchdb@node1 <0.10417.1> -------- Bad security object in <<"test2">>: [{{[{<<"members">>,{[{<<"roles">>,[<<"_admin">>]}]}},{<<"admins">>,{[{<<"roles">>,[<<"_admin">>]}]}}]},1},{{[{<<"admins">>,{[{<<"names">>,[<<"superuser">>]},{<<"roles">>,[<<"admins">>]}]}},{<<"members">>,{[{<<"names">>,[<<"user1">>,<<"user2">>]},{<<"roles">>,[<<"developers">>]}]}}]},1}]
```
This was discussed with @janl and @rnewson on Slack at https://couchdb.slack.com/archives/C49LEE7NW/p1628257123045300 which can be helpful.
Contributor guide
Research direction
Start by reproducing the CouchDB 3.1.1 scenario with a three-node cluster, q=1, n=2, a blank re-added node, and the test2 _security document. Trace the _security and _sync_shards entry points while comparing responses and logs across nodes; done means the configured security object remains consistent after resync and restart without the reported Bad security object error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100