cockroachdb / cockroachdb/cockroach

server: /debug/lsm-viz panics with "addFileToSublevels found existing newer file"

Open
#170,896 1 comment 0 reactions 0 assignees View on GitHub
A-storage C-bug O-agent T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Hitting `/debug/lsm-viz/` reliably 500s with a panic when the underlying LSM contains certain file shapes (observed: linked external files placed at intermediate levels with seqnums that interact with L0 files).

**Reproduction context**

Observed three consecutive times on a cluster after an online-restore link phase had completed and L0 was holding ~137 GB of stacked external-backed SSTs. The cluster had `debug.panic_on_failed_assertions=true`.

To reproduce on a fresh cluster: restore `gs://cockroach-fixtures-us-east1/roachtest/master/tpcc-5k/20260522-090958.790` `AS OF SYSTEM TIME '2026-05-22T10:56:00Z'` `WITH EXPERIMENTAL DEFERRED COPY, UNSAFE_RESTORE_INCOMPATIBLE_VERSION`, pause the download job at `restore.before_download`, then hit `/debug/lsm-viz/1`.

**Stack**

```
addFileToSublevels found existing newer file
pebble/internal/manifest/l0_sublevels.go:679 (addFileToSublevels)
pebble/internal/manifest/l0_sublevels.go:309 (newL0Sublevels)
pebble/internal/manifest/l0_sublevels.go:2237 (L0Organizer.ResetForTesting)
pebble/internal/manifest/version.go:124 (NewVersionForTesting)
pebble/tool/lsm.go:326 (lsmT.buildEdits)
pebble/tool/lsm.go:160 (lsmT.runLSM)
pkg/server/debug/server.go:228 (analyzeLSM)
pkg/server/debug/server.go:285 (Server.RegisterEngines.func2)
```

The path goes through `pebble/tool` and uses `NewVersionForTesting` / `ResetForTesting` to reconstruct an L0 organizer from the current LSM state. The assertion fires when sublevel reconstruction encounters a file ordering it considers invalid. Production-path code that maintains the L0 organizer incrementally doesn't trip this; only the from-scratch reconstruction in the viz helper does.

**Code references**

- [pkg/server/debug/server.go:228](https://github.com/cockroachdb/cockroach/blob/master/pkg/server/debug/server.go#L228) — `analyzeLSM`
- pebble (external): `internal/manifest/l0_sublevels.go:679` `addFileToSublevels`
- pebble (external): `internal/manifest/l0_sublevels.go:2237` `L0Organizer.ResetForTesting`
- pebble (external): `tool/lsm.go:160` `lsmT.runLSM`

**Investigations remaining**

- Capture a minimal `MANIFEST` excerpt that triggers it for fixture-based regression coverage.
- Determine whether the assertion is a real Pebble invariant being violated by the production-path code, or an over-broad check in the test-helper reconstruction path. If the latter, relaxing the test-helper assertion (or routing `analyzeLSM` around it) is probably the right fix.

Test coverage gap that allowed this to ship is tracked separately.

Upstream pebble issue: cockroachdb/pebble#6075.

Related: #170225.

Jira issue: CRDB-64225

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.