dolt fsck panics instead of reporting damage when a table file's index is broken
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 120
Description
Thanks again for the quick root cause and fix in #11747 / #11757. While checking things after upgrading, we ran into a second, smaller problem: `dolt fsck` crashes on the damaged database from #11747 instead of reporting the damage.
### What happens
That database has a large table file that was written with a broken index (the mechanism fixed in #11757). Running `dolt fsck` in a copy of it panics:
- **dolt 2.2.3:** `panic: runtime error: slice bounds out of range [:22439618760] with capacity 2506448`
- **dolt 2.3.3:** the identical panic in the same frames, exit status 2. Full output:
```
panic: runtime error: slice bounds out of range [:22439618760] with capacity 2506448
goroutine 1 [running]:
github.com/dolthub/dolt/go/store/nbs.onHeapTableIndex.indexEntry({{0x4272e00, 0x16b3e97e29e0}, 0x16b3e9994840, {0x16b3e9c3a000, 0x263ebc, 0x593d20}, {0x16b3ea1ce000, 0xcbf98, 0xcbf98}, {0x16b3e9e9debc, ...}, ...}, ...)
github.com/dolthub/dolt/go/store/nbs/table_index.go:298 +0x1da
github.com/dolthub/dolt/go/store/nbs.tableReader.tolerantIterateAllChunks({{0x42f9150, 0x16b3e99ae160}, {0x425b6f0, 0x16b3e942d380}, 0x1000, {0x16b3ea29a000, 0x32fe5, 0x32fe5}, 0x16b3e9994844, 0x16b3e99aa078}, ...)
github.com/dolthub/dolt/go/store/nbs/table_reader.go:894 +0x14f
github.com/dolthub/dolt/go/store/nbs.(*NomsBlockStore).TolerantIterateAllChunks(0x16b3e9726ea0, {0x425bab0, 0x16b3e974fa00}, 0x16b3e99ac040, 0x16b3e99ac050)
github.com/dolthub/dolt/go/store/nbs/store.go:2697 +0x3a7
github.com/dolthub/dolt/go/store/nbs.(*GenerationalNBS).TolerantIterateAllChunks(0x16b3e99aa048, {0x425bab0, 0x16b3e974fa00}, 0x16b3e99ac040, 0x16b3e99ac050)
github.com/dolthub/dolt/go/store/nbs/generational_chunk_store.go:596 +0x34
github.com/dolthub/dolt/go/cmd/dolt/commands.(*roundTripper).scanAll(0x16b3e99921e0, {0x425bab0, 0x16b3e974fa00})
github.com/dolthub/dolt/go/cmd/dolt/commands/fsck.go:531 +0xe5
github.com/dolthub/dolt/go/cmd/dolt/commands.fsckOnChunkStore({0x425bab0, 0x16b3e974fa00}, 0x16b3e99aa048, 0x16b3e97c9810, 0x16b3e97c3960)
github.com/dolthub/dolt/go/cmd/dolt/commands/fsck.go:421 +0x134
github.com/dolthub/dolt/go/cmd/dolt/commands.FsckCmd.Exec.func3(0x4310d50?, {0x425bab0, 0x16b3e974fa00}, 0x16b3e9427b90?, 0x425bab0?)
github.com/dolthub/dolt/go/cmd/dolt/commands/fsck.go:167 +0x6a
github.com/dolthub/dolt/go/cmd/dolt/commands.FsckCmd.Exec({}, {0x425bab0, 0x16b3e974fa00}, {0x16b3e97e29a0, 0x9}, {0x16b3e8d40070, 0x0, 0x0}, 0x16b3e97c79a0, {0x0, ...})
github.com/dolthub/dolt/go/cmd/dolt/commands/fsck.go:182 +0x6c5
github.com/dolthub/dolt/go/cmd/dolt/cli.SubCommandHandler.handleCommand({{0x3bb093e, 0x4}, {0x3bd5f3d, 0x11}, {0x0, 0x0}, {0x66af6a0, 0x39, 0x39}, 0x0}, ...)
github.com/dolthub/dolt/go/cmd/dolt/cli/command.go:255 +0x456
github.com/dolthub/dolt/go/cmd/dolt/cli.SubCommandHandler.Exec({{0x3bb093e, 0x4}, {0x3bd5f3d, 0x11}, {0x0, 0x0}, {0x66af6a0, 0x39, 0x39}, 0x0}, ...)
github.com/dolthub/dolt/go/cmd/dolt/cli/command.go:210 +0x309
main.runMain()
github.com/dolthub/dolt/go/cmd/dolt/dolt.go:572 +0x2bb8
main.main()
github.com/dolthub/dolt/go/cmd/dolt/dolt.go:226 +0x36
```
### Where it seems to come from
`tolerantIterateAllChunks` calls `tr.idx.indexEntry(i, &h)` and already routes per-entry errors to `errCb`. But at v2.3.3, `onHeapTableIndex.indexEntry` (table_index.go:298) slices `ti.suffixes[o : o+hash.SuffixLen]` with the ordinal from the prefix tuple and never checks it against the chunk count. The broken file is `o4ojdar03lipd8m3lmvh6cjqmj37anvu` (333,447,188 bytes). Its manifest entry lists 208,869 chunks, so `suffixes` is 208,869 × 12 = 2,506,428 bytes, while the ordinal read back is 1,869,968,229. A bounds check on the ordinal, and probably on the offsets it leads to, would let fsck report the file through `errCb` instead of crashing.
### Expected
fsck reports that this table file's index is corrupt, checks whatever else it can, and exits non-zero. A damaged store is exactly when someone reaches for fsck.
### Environment and data
Linux x86_64, official release binaries for 2.2.3 and 2.3.3, run against a byte-identical copy of the database. The maintainer on #11747 has had access to this database, and we're happy to share it again on request.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.