codenotary / codenotary/immudb

Transient S3 error during open leaves dbRef{db:nil} cached; GetState returns "unable to get state" until restart

Open
#2,155 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
9k
Forks
378
Avg merge
3d 1h
Merged PRs (30d)
3

Description

**What happened**

A transient S3 `403` during a new database's first open leaves a `dbRef{db: nil}` cached in `DBManager.dbCache`. `GetState` then fails on every call, the server keeps reporting healthy, and only a restart recovered it.

Creating `audit` and `billing` against S3 storage, 3 of 53 S3 calls returned `403` during their first open. Every other call in that window, same bucket and credentials, returned `200`, and the failing key succeeded one second later — provider flapping, not auth:

```
S3 PUT ...//audit/aht/commit/00000000.di failed with status code 403 (403 Forbidden)
S3 PUT ...//audit/aht/commit/00000000.di 200 OK <-- same key, one second later

{"caller":"/src/pkg/database/database.go:1739","level":"error",
"message":"unable to open database: ... 403 (403 Forbidden): could not open indexer"}
```

`billing` failed at 12:06:42, `audit` at 12:09:15. Until the restart at 15:29:52 the log held nothing but this, once per minute per database:

```
"error getting current state of db audit to update the number of entries metric: unable to get state"
```

After the restart all four databases opened and 74/74 S3 calls returned `200`.

**What you expected to happen**

A transient remote-storage error during an open should not be terminal. The failed open should clean up so a later access retries, or retry with backoff.

**How to reproduce it (as minimally and precisely as possible)**

A real provider fault, not a synthetic one, so these are conditions rather than a script:

1. immudb with `S3_STORAGE=true`.
2. A proxy in front of the endpoint failing a small share of requests with `403`.
3. Call `CreateDatabaseV2` — `databaseList.Put` installs a `lazyDB`, so the open runs on first access.
4. Land one injected `403` inside that open.
5. Expect `unable to open database` once, then `unable to get state` per minute. Ours never recovered until we restarted 3 h 20 min later.

**Environment**

```shell
immudb 1.11.0
Commit : bfdce03649f52d575be46f74425fd18eaf4fa69c
Built at: Thu, 03 Sep 2026 06:46:12 UTC
S3 storage, endpoint https://fsn1.your-objectstorage.com (Hetzner Object Storage, Ceph RGW)
Auth enabled: true Synced mode: true Log format: json
```

**Additional info (any other context about the problem)**

[`allocDB`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L206) caches the `dbRef` before the open is attempted. When `openDB` fails, [`Get`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L183) calls [`Release`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L236), which zeroes the refcount but leaves the entry in `dbCache` — a `dbRef{db: nil, count: 0}`.

The file already knows this state exists. [`CloseAll`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L498):

```go
// ref.db may be nil if the database was never successfully opened
// (e.g. openDB failed in Get and left a db-less ref in the cache).
if ref.db != nil {
```

[`GetState`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L336) calls the same state impossible:

```go
if dbRef := ref.(*dbRef); dbRef != nil && dbRef.db != nil {
return dbRef.db.CurrentState()
}
// this condition should never happen
return nil, fmt.Errorf("unable to get state")
```

The cached entry makes that branch permanent — `GetState` never reaches the `m.Get` fallback below it that would retry the open. [`IsActive`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L514) is wrong for the same reason: it only checks that the cache lookup succeeds, so it reports a database that never opened as active.

**What I could not explain**

`Get` *should* retry `openDB` on the next access — the cached ref has `db == nil`, and a failed open never sets `dbInfo.closed`, so `allocDB` does not short-circuit. Yet our clients retried for over three hours without connecting, and no S3 traffic appeared in the log during that window, so nothing was re-driving the open. The cached ref and the stuck `GetState` are established; why no caller re-drove the open is not. I would rather flag that than guess.

**Suggested fix**

Pop the entry from `dbCache` on the `openDB` error path in `Get`, so a `dbRef` with a nil `db` is never observable. That also makes `GetState`'s comment true again.

**Related**

#1997 — same symptom on an S3 backend, no reproduction; the workaround comment in `DatabaseListV2` cites it. Possibly the same root cause with a concrete trigger.
#2156 — this incident stayed invisible because `Health` cannot report a database that failed to open.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/database/db_manager.go at allocDB, Get, GetState, and IsActive, following how dbCache entries are created, released, and reused after openDB fails. Confirm the failure path cannot leave a nil database reference cached, and that later access can retry while GetState and IsActive no longer remain stuck on the failed-open state.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.