influxdata / influxdata/influxdb

InfluxDB 3 Enterprise: unclean shutdown leaves zero-byte V2 compaction checkpoints, causing a permanent unrecoverable startup loop (and silent orphaning of all history if deleted)

Open
#27,624 1 comment 0 reactions 0 assignees View on GitHub
v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

## Summary

After an unclean host shutdown, InfluxDB 3 Enterprise can be left with **zero-length**
V2 compaction checkpoint files (`*.ptv2chk.lz4`). On every subsequent start the server
tries to lz4-decompress one, fails, retries with backoff, and exits 1 — permanently.

There is no recovery path that preserves data. The only way to make the server boot is
to delete the checkpoints, and doing that resets the compactor to
`loaded_checkpoint_sequence=0`, which orphans every compacted parquet file on disk. The
server then comes up perfectly healthy and returns **0 rows** for every measurement
while hundreds of MB of parquet sit unreferenced.

I think there are two separate defects here:

1. **Durability** — checkpoints do not appear to be written atomically (temp file →
`fsync` → `rename`) before being treated as live. A zero-length checkpoint should
never be observable on disk.
2. **Recovery** — a truncated/undecodable checkpoint is a hard, infinitely-retried
error. There is no fallback to an older valid checkpoint, and no way to rebuild
compaction state from the manifest/parquet set.

## Environment

- InfluxDB 3 **Enterprise** (At-Home license), via the `erik73/hassio-addons` InfluxDB3
app v1.1.3, image `ghcr.io/erik73/influxdb3/amd64`
- Home Assistant OS 18.2, amd64 / generic-x86-64
- Data dir on ext4 mounted with `commit=30` (Home Assistant OS default for `/mnt/data`),
so an unclean shutdown can lose up to 30s of un-`fsync`'d writes.

## What happened

The host rebooted uncleanly at `2026-09-10 22:02 UTC`. That left **12 zero-byte files**,
every one stamped `22:02:xx`:

```
influxdb3/cluster0/cv2/checkpoints/00000000000000219446.ptv2chk.lz4 <- fatal
influxdb3/cluster0/cv2/checkpoints/00000000000000219447.ptv2chk.lz4 <- fatal
influxdb3/cluster0/cv2/checkpoints/00000000000000219448.ptv2chk.lz4 <- fatal
influxdb3//cluster0/cv2/windows/2026-09-10/p0/run_sets/*.idx (2)
influxdb3//cluster0/cv2/windows/2026-09-10/p0/files/*.pt (2)
influxdb3//0/pt_wal/*.pt (4)
influxdb3//0/gen0/*.pt (2)
influxdb3/0/pt_snapshots/*.ptsnap (2)
```

Critically, **all three** checkpoints were zero-length — there was no older intact one
to fall back to.

Startup then loops forever (1/2/4/8/10s backoff, then exit 1):

```
INFO influxdb3::commands::serve: temp catalog loaded (484 ms)
INFO influxdb3::commands::serve: catalog initialized with uuid: ''
INFO influxdb3::commands::serve: licensing initialized (724 ms)
WARN influxdb3_lib::commands::serve: server initialization failed with retryable error, retrying
error=failed to initialize write buffer: Failed to create V2 compactor:
internal combined mode error: failed to start standby compaction consumer:
Internal error: Failed to decompress V2 checkpoint: expected another byte, found none
retry_in_secs=1.0
...
Serve command failed: failed to initialize write buffer: Failed to create V2 compactor:
internal combined mode error: failed to start standby compaction consumer:
Internal error: Failed to decompress V2 checkpoint: expected another byte, found none
```

Note the catalog itself loads fine — only the compactor is wedged.

## The data-loss trap

Deleting the empty checkpoints makes the server start, which looks like a fix. It is not:

```
WARN influxdb3_pacha_tree::enterprise::compactor::compactor_loop::checkpoint:
compactor delete queue is newer than the loaded checkpoint (likely a restore);
discarding it to avoid deleting files the restored checkpoint references
queue_checkpoint_sequence=219448 loaded_checkpoint_sequence=0
```

`loaded_checkpoint_sequence=0`. The server reports healthy, `/health` returns `OK`,
738MB of parquet is still on disk — and `SELECT count(*)` returns 0 for every
measurement. The data is intact but unreachable.

Removing the checkpoints also exposed a second truncation fault behind them, from the
zero-byte WAL/snapshot/gen0 files:

```
WARN influxdb3_lib::commands::serve: server initialization failed with retryable error, retrying
error=failed to initialize write buffer: PachaTree error: Internal error: Manifest too small for magic
```

So the fragility isn't limited to checkpoints — zero-length segments elsewhere in the
tree are also treated as hard errors rather than being skipped.

## What actually recovered the data

A restore of the whole data directory from a backup predating the reboot. History came
back in full (384,996 rows in a single measurement, Aug 24 → present). Everything
written between the backup and the crash was unrecoverable.

## Suggested fixes

- Write checkpoints atomically: temp file → `fsync` → `rename`. A zero-length
checkpoint should never be observable.
- On a truncated/empty/undecodable checkpoint, fall back to the newest **valid** older
checkpoint rather than failing.
- If no valid checkpoint exists, fail loudly and *distinctly*, and do not leave
"delete the checkpoint files" as the only apparent remedy — it silently orphans all
compacted history. An explicit rebuild-compaction-state-from-manifest path would turn
this from data loss into an inconvenience.
- Skip or truncate zero-length WAL / snapshot / gen0 segments on replay instead of
erroring.

Contributor guide

Open the contributing guide

Research direction

Start with the V2 compactor checkpoint handling referenced by the startup trace, then trace PachaTree replay for the WAL, snapshot, and gen0 truncation errors. Reproduce startup using the reported zero-byte files and determine how atomic writes, valid-checkpoint fallback or rebuild, and safe segment handling should be verified. Done means an unclean shutdown cannot silently orphan compacted history.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.