microsoft / microsoft/garnet

Key becomes unreachable after recovery when its record was reused via revivification

Open
#2,127 0 comments 0 reactions 1 assignee Claimed by @TedHartMS View on GitHub
Dominant language
C#
Stars
12k
Forks
703
Avg merge
2d 19h
Merged PRs (30d)
36

Description

### Describe the bug

After a restart, a key can end up in a state where `KEYS` and `SCAN` enumerate it and `DBSIZE` counts it, but `EXISTS` returns 0 and `GET` returns nil.

It happens when the key's record was reused via revivification and a checkpoint was taken afterwards:

- write a key
- delete the key
- checkpoint
- write key again
- checkpoint
- restart Garnet
- get key returns nil

So, reproducible after five commands from the CLI and a restart.

### Steps to reproduce the bug

### Environment

- Garnet **2.1.6**, 64-bit, standalone mode
- Persistence enabled with revivification on (config below)

### Configuration

```json
{
"EnableAOF": true,
"EnableRevivification": true,
"Recover": true,
"CheckpointDir": "C:/Program Files/Garnet/checkpoints",
}
```

The following steps conducted with an empty cache:

### Reproduction

```
> SET a 1
"OK"

> DEL a
(integer) 1

> SAVE
"OK"

> SET a 2
"OK"

> SAVE
"OK"

> EXISTS a
(integer) 1
```

Restart Garnet, then:

```
> EXISTS a
(integer) 0

> GET a
(nil)

> KEYS *
1) "a"

> DBSIZE
(integer) 1
```

`EXISTS` returns 1 before the restart and 0 after it. `KEYS` and `DBSIZE` disagree with `EXISTS` and `GET` on whether the key exists.

Writing the key again repairs it. So, a subsequent `SET a 3` followed by `GET a` returns the new value normally.

---

### Reviv Info shows the record was reused

Immediately after `SET a 2` in the repro sequence above:

```
> INFO STOREREVIV
# StoreDeletedRecordRevivification_DB_0
Successful Adds: 1
Failed Adds: 0
Successful Takes: 1
Failed Takes: 0
Empty bins: 0
Address limit: 0
Record size limit: 0
Successful In-Chain: 0
Failed In-Chain: 0
```

`Successful Adds: 1` from the `DEL` and `Successful Takes: 1` from the `SET`, so the new record occupies the freed record via the free record pool.

---

**The key also survives the restart if you don't do the second `SAVE`** (after the re-add); as follows:

```
SET a 1
DEL a
SAVE
SET a 2
[no checkpoint here]
>>> restart <
> EXISTS a
(integer) 1
```

So recovery from AOF replay alone reconstructs the key correctly. The problem appears only when a checkpoint is taken after the record has been reused.

---

### Also observed with two different keys

The same issue occurs when the reused record belongs to a different key:

```
SET a 1
DEL a
SAVE
SET b 2
SAVE
>>> restart <
> EXISTS b
(integer) 0
> KEYS *
1) "b"
```

### Expected behavior

After recovery, a key that existed at checkpoint time is reachable by `GET`, `EXISTS`, and other point lookups, consistent with `KEYS`, `SCAN`, and `DBSIZE`.

### Screenshots

_No response_

### Release version

v2.1.6

### IDE

_No response_

### OS version

Windows Server 2025

### Additional context

In our actual deployment AofSizeLimit is configured, so checkpoints fire automatically every few minutes, and the workload deletes and recreates keys continuously, so records are reused constantly. Any restart in that state leaves keys unreachable, and so we hit this issue routinely while testing recovery. We had assumed it was a consequence of #2101, but the repro steps above show these are distinct issues.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.