cockroachdb / cockroachdb/cockroach
server: /debug/lsm-viz panics with "addFileToSublevels found existing newer file"
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Hitting `/debug/lsm-viz/` reliably 500s with a panic when the underlying LSM contains certain file shapes (observed: linked external files placed at intermediate levels with seqnums that interact with L0 files).
**Reproduction context**
Observed three consecutive times on a cluster after an online-restore link phase had completed and L0 was holding ~137 GB of stacked external-backed SSTs. The cluster had `debug.panic_on_failed_assertions=true`.
To reproduce on a fresh cluster: restore `gs://cockroach-fixtures-us-east1/roachtest/master/tpcc-5k/20260522-090958.790` `AS OF SYSTEM TIME '2026-05-22T10:56:00Z'` `WITH EXPERIMENTAL DEFERRED COPY, UNSAFE_RESTORE_INCOMPATIBLE_VERSION`, pause the download job at `restore.before_download`, then hit `/debug/lsm-viz/1`.
**Stack**
```
addFileToSublevels found existing newer file
pebble/internal/manifest/l0_sublevels.go:679 (addFileToSublevels)
pebble/internal/manifest/l0_sublevels.go:309 (newL0Sublevels)
pebble/internal/manifest/l0_sublevels.go:2237 (L0Organizer.ResetForTesting)
pebble/internal/manifest/version.go:124 (NewVersionForTesting)
pebble/tool/lsm.go:326 (lsmT.buildEdits)
pebble/tool/lsm.go:160 (lsmT.runLSM)
pkg/server/debug/server.go:228 (analyzeLSM)
pkg/server/debug/server.go:285 (Server.RegisterEngines.func2)
```
The path goes through `pebble/tool` and uses `NewVersionForTesting` / `ResetForTesting` to reconstruct an L0 organizer from the current LSM state. The assertion fires when sublevel reconstruction encounters a file ordering it considers invalid. Production-path code that maintains the L0 organizer incrementally doesn't trip this; only the from-scratch reconstruction in the viz helper does.
**Code references**
- [pkg/server/debug/server.go:228](https://github.com/cockroachdb/cockroach/blob/master/pkg/server/debug/server.go#L228) — `analyzeLSM`
- pebble (external): `internal/manifest/l0_sublevels.go:679` `addFileToSublevels`
- pebble (external): `internal/manifest/l0_sublevels.go:2237` `L0Organizer.ResetForTesting`
- pebble (external): `tool/lsm.go:160` `lsmT.runLSM`
**Investigations remaining**
- Capture a minimal `MANIFEST` excerpt that triggers it for fixture-based regression coverage.
- Determine whether the assertion is a real Pebble invariant being violated by the production-path code, or an over-broad check in the test-helper reconstruction path. If the latter, relaxing the test-helper assertion (or routing `analyzeLSM` around it) is probably the right fix.
Test coverage gap that allowed this to ship is tracked separately.
Upstream pebble issue: cockroachdb/pebble#6075.
Related: #170225.
Jira issue: CRDB-64225
Contributor guide
Assessment
This issue has not been assessed yet.