apache / apache/couchdb

In 3 node cluster (with q=1), when any one node fails and is re-added as blank, _security object is reset to default

Open
#3,696 0 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

## Your Environment

```
{
"couchdb": "Welcome",
"version": "3.1.1",
"git_sha": "ce596c65d",
"uuid": "8d406054df5edac06ee4906f3259e62f",
"features": [
"access-ready",
"partitioned",
"pluggable-storage-engines",
"reshard",
"scheduler"
],
"vendor": {
"name": "The Apache Software Foundation"
}
}
```

## Description

I have a 3 node couchdb `3.1.1` cluster with following configuration:

```
[cluster]
q=1
n=2
```

There is a non-partitioned database named `test2`, whose shards resides on `node1` and `node2`. `test2` database has following cluster settings:

```
"cluster":{"q":1,"n":2,"w":2,"r":2}
```

`test2` database has few documents and it's `_security` is:

```
{"admins":{"names":["superuser"],"roles":["admins"]},"members":{"names":["user1","user2"],"roles":["developers"]}}
```

I'm now running a scenario where any of the node's disk crashes. Let's say `node2`'s disk crashes.

I have performed following steps:

- Remove `node2` from the cluster
- Replace `node2` disk with a new blank disk
- Start `node2`
- Add `node2` into cluster
- It would resync the shards as its blank
- After a while, resync is completed

> At this stage, `test2` database's shards file(test2.1628258896.couch) can be seen on `node2`.

### 1: Now, when I retrieve `_security` for `test2` database from `node2`, it is reset to the default like:

```
{"members":{"roles":["_admin"]},"admins":{"roles":["_admin"]}}
```

If I retrieve `_security` from `node1` or `node3`, it responds with the correct: (which I set earlier before `node2` crash)

```
{"admins":{"names":["superuser"],"roles":["admins"]},"members":{"names":["user1","user2"],"roles":["developers"]}}
```

### 2: When I restart all nodes, there are following error logs shown:

```
node2 | [error] 2021-08-06T12:59:21.842335Z couchdb@node2 <0.4465.0> -------- Bad security object in <<"test2">>: [{{[{<<"members">>,{[{<<"roles">>,[<<"_admin">>]}]}},{<<"admins">>,{[{<<"roles">>,[<<"_admin">>]}]}}]},1},{{[{<<"admins">>,{[{<<"names">>,[<<"superuser">>]},{<<"roles">>,[<<"admins">>]}]}},{<<"members">>,{[{<<"names">>,[<<"user1">>,<<"user2">>]},{<<"roles">>,[<<"developers">>]}]}}]},1}]
```

### 3: Not relevant to crash, but relevant to same `_security` thing.

When 1 node is down, `PUT _security` fails with: `{"error":"error","reason":"no_majority"}`

- `node2` is down
- Create a new DB `test3`, whose shards would reside on `node1` and `node2`
- `PUT _security` for `test3`, it would fail with:

```
{"error":"error","reason":"no_majority"}
```

## Other observations
- `_sync_shards` throws same `Bad security object` error
- It works for `q=2`
- It fixes the issue when I updated `_security` of `test2` DB again

#### _sync_shards logs

```
node1 | [notice] 2021-08-06T13:38:55.064067Z couchdb@node1 <0.10379.1> c89bd8ac41 localhost:5984 172.20.0.1 admin POST /test2/_sync_shards 202 ok 2
node2 | [error] 2021-08-06T13:38:55.080064Z couchdb@node2 <0.7232.0> -------- Bad security object in <<"test2">>: [{{[{<<"members">>,{[{<<"roles">>,[<<"_admin">>]}]}},{<<"admins">>,{[{<<"roles">>,[<<"_admin">>]}]}}]},1},{{[{<<"admins">>,{[{<<"names">>,[<<"superuser">>]},{<<"roles">>,[<<"admins">>]}]}},{<<"members">>,{[{<<"names">>,[<<"user1">>,<<"user2">>]},{<<"roles">>,[<<"developers">>]}]}}]},1}]
node1 | [error] 2021-08-06T13:38:55.080503Z couchdb@node1 <0.10417.1> -------- Bad security object in <<"test2">>: [{{[{<<"members">>,{[{<<"roles">>,[<<"_admin">>]}]}},{<<"admins">>,{[{<<"roles">>,[<<"_admin">>]}]}}]},1},{{[{<<"admins">>,{[{<<"names">>,[<<"superuser">>]},{<<"roles">>,[<<"admins">>]}]}},{<<"members">>,{[{<<"names">>,[<<"user1">>,<<"user2">>]},{<<"roles">>,[<<"developers">>]}]}}]},1}]
```

This was discussed with @janl and @rnewson on Slack at https://couchdb.slack.com/archives/C49LEE7NW/p1628257123045300 which can be helpful.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the CouchDB 3.1.1 scenario with a three-node cluster, q=1, n=2, a blank re-added node, and the test2 _security document. Trace the _security and _sync_shards entry points while comparing responses and logs across nodes; done means the configured security object remains consistent after resync and restart without the reported Bad security object error.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.