Key becomes unreachable after recovery when its record was reused via revivification
- Dominant language
- C#
- Stars
- 12k
- Forks
- 703
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 36
Description
### Describe the bug
After a restart, a key can end up in a state where `KEYS` and `SCAN` enumerate it and `DBSIZE` counts it, but `EXISTS` returns 0 and `GET` returns nil.
It happens when the key's record was reused via revivification and a checkpoint was taken afterwards:
- write a key
- delete the key
- checkpoint
- write key again
- checkpoint
- restart Garnet
- get key returns nil
So, reproducible after five commands from the CLI and a restart.
### Steps to reproduce the bug
### Environment
- Garnet **2.1.6**, 64-bit, standalone mode
- Persistence enabled with revivification on (config below)
### Configuration
```json
{
"EnableAOF": true,
"EnableRevivification": true,
"Recover": true,
"CheckpointDir": "C:/Program Files/Garnet/checkpoints",
}
```
The following steps conducted with an empty cache:
### Reproduction
```
> SET a 1
"OK"
> DEL a
(integer) 1
> SAVE
"OK"
> SET a 2
"OK"
> SAVE
"OK"
> EXISTS a
(integer) 1
```
Restart Garnet, then:
```
> EXISTS a
(integer) 0
> GET a
(nil)
> KEYS *
1) "a"
> DBSIZE
(integer) 1
```
`EXISTS` returns 1 before the restart and 0 after it. `KEYS` and `DBSIZE` disagree with `EXISTS` and `GET` on whether the key exists.
Writing the key again repairs it. So, a subsequent `SET a 3` followed by `GET a` returns the new value normally.
---
### Reviv Info shows the record was reused
Immediately after `SET a 2` in the repro sequence above:
```
> INFO STOREREVIV
# StoreDeletedRecordRevivification_DB_0
Successful Adds: 1
Failed Adds: 0
Successful Takes: 1
Failed Takes: 0
Empty bins: 0
Address limit: 0
Record size limit: 0
Successful In-Chain: 0
Failed In-Chain: 0
```
`Successful Adds: 1` from the `DEL` and `Successful Takes: 1` from the `SET`, so the new record occupies the freed record via the free record pool.
---
**The key also survives the restart if you don't do the second `SAVE`** (after the re-add); as follows:
```
SET a 1
DEL a
SAVE
SET a 2
[no checkpoint here]
>>> restart <
> EXISTS a
(integer) 1
```
So recovery from AOF replay alone reconstructs the key correctly. The problem appears only when a checkpoint is taken after the record has been reused.
---
### Also observed with two different keys
The same issue occurs when the reused record belongs to a different key:
```
SET a 1
DEL a
SAVE
SET b 2
SAVE
>>> restart <
> EXISTS b
(integer) 0
> KEYS *
1) "b"
```
### Expected behavior
After recovery, a key that existed at checkpoint time is reachable by `GET`, `EXISTS`, and other point lookups, consistent with `KEYS`, `SCAN`, and `DBSIZE`.
### Screenshots
_No response_
### Release version
v2.1.6
### IDE
_No response_
### OS version
Windows Server 2025
### Additional context
In our actual deployment AofSizeLimit is configured, so checkpoints fire automatically every few minutes, and the workload deletes and recreates keys continuously, so records are reused constantly. Any restart in that state leaves keys unreachable, and so we hit this issue routinely while testing recovery. We had assumed it was a consequence of #2101, but the repro steps above show these are distinct issues.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.