codenotary / codenotary/immudb
Transient S3 error during open leaves dbRef{db:nil} cached; GetState returns "unable to get state" until restart
- Dominant language
- Go
- Stars
- 9k
- Forks
- 378
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 3
Description
**What happened**
A transient S3 `403` during a new database's first open leaves a `dbRef{db: nil}` cached in `DBManager.dbCache`. `GetState` then fails on every call, the server keeps reporting healthy, and only a restart recovered it.
Creating `audit` and `billing` against S3 storage, 3 of 53 S3 calls returned `403` during their first open. Every other call in that window, same bucket and credentials, returned `200`, and the failing key succeeded one second later — provider flapping, not auth:
```
S3 PUT ...//audit/aht/commit/00000000.di failed with status code 403 (403 Forbidden)
S3 PUT ...//audit/aht/commit/00000000.di 200 OK <-- same key, one second later
{"caller":"/src/pkg/database/database.go:1739","level":"error",
"message":"unable to open database: ... 403 (403 Forbidden): could not open indexer"}
```
`billing` failed at 12:06:42, `audit` at 12:09:15. Until the restart at 15:29:52 the log held nothing but this, once per minute per database:
```
"error getting current state of db audit to update the number of entries metric: unable to get state"
```
After the restart all four databases opened and 74/74 S3 calls returned `200`.
**What you expected to happen**
A transient remote-storage error during an open should not be terminal. The failed open should clean up so a later access retries, or retry with backoff.
**How to reproduce it (as minimally and precisely as possible)**
A real provider fault, not a synthetic one, so these are conditions rather than a script:
1. immudb with `S3_STORAGE=true`.
2. A proxy in front of the endpoint failing a small share of requests with `403`.
3. Call `CreateDatabaseV2` — `databaseList.Put` installs a `lazyDB`, so the open runs on first access.
4. Land one injected `403` inside that open.
5. Expect `unable to open database` once, then `unable to get state` per minute. Ours never recovered until we restarted 3 h 20 min later.
**Environment**
```shell
immudb 1.11.0
Commit : bfdce03649f52d575be46f74425fd18eaf4fa69c
Built at: Thu, 03 Sep 2026 06:46:12 UTC
S3 storage, endpoint https://fsn1.your-objectstorage.com (Hetzner Object Storage, Ceph RGW)
Auth enabled: true Synced mode: true Log format: json
```
**Additional info (any other context about the problem)**
[`allocDB`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L206) caches the `dbRef` before the open is attempted. When `openDB` fails, [`Get`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L183) calls [`Release`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L236), which zeroes the refcount but leaves the entry in `dbCache` — a `dbRef{db: nil, count: 0}`.
The file already knows this state exists. [`CloseAll`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L498):
```go
// ref.db may be nil if the database was never successfully opened
// (e.g. openDB failed in Get and left a db-less ref in the cache).
if ref.db != nil {
```
[`GetState`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L336) calls the same state impossible:
```go
if dbRef := ref.(*dbRef); dbRef != nil && dbRef.db != nil {
return dbRef.db.CurrentState()
}
// this condition should never happen
return nil, fmt.Errorf("unable to get state")
```
The cached entry makes that branch permanent — `GetState` never reaches the `m.Get` fallback below it that would retry the open. [`IsActive`](https://github.com/codenotary/immudb/blob/bfdce03649f52d575be46f74425fd18eaf4fa69c/pkg/database/db_manager.go#L514) is wrong for the same reason: it only checks that the cache lookup succeeds, so it reports a database that never opened as active.
**What I could not explain**
`Get` *should* retry `openDB` on the next access — the cached ref has `db == nil`, and a failed open never sets `dbInfo.closed`, so `allocDB` does not short-circuit. Yet our clients retried for over three hours without connecting, and no S3 traffic appeared in the log during that window, so nothing was re-driving the open. The cached ref and the stuck `GetState` are established; why no caller re-drove the open is not. I would rather flag that than guess.
**Suggested fix**
Pop the entry from `dbCache` on the `openDB` error path in `Get`, so a `dbRef` with a nil `db` is never observable. That also makes `GetState`'s comment true again.
**Related**
#1997 — same symptom on an S3 backend, no reproduction; the workaround comment in `DatabaseListV2` cites it. Possibly the same root cause with a concrete trigger.
#2156 — this incident stayed invisible because `Health` cannot report a database that failed to open.
Contributor guide
Research direction
Start in pkg/database/db_manager.go at allocDB, Get, GetState, and IsActive, following how dbCache entries are created, released, and reused after openDB fails. Confirm the failure path cannot leave a nil database reference cached, and that later access can retry while GetState and IsActive no longer remain stuck on the failed-open state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100