cockroachdb / cockroachdb/pebble
CRC32C mismatch reported for L6 SST, but standalone sstable check passes and file size matches manifest
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
## Summary
I'm reporting an on-disk Pebble corruption encountered through Kubo/IPFS using `pebbleds`.
During `ipfs repo gc`, the operation started removing blocks normally and then failed with:
```text
Error: unexpected EOF
```
Subsequent attempts returned:
```text
Error: EOF
```
Further verification identified an SST with a CRC32C checksum mismatch:
```text
67232266.sst
pebble: corruption
crc32c checksum mismatch
```
I preserved the complete affected repository before rebuilding the Kubo node and have run additional diagnostics using the Pebble CLI version matching the Pebble version used by Kubo (`v2.1.6`).
The interesting part is that the SST still has exactly the size recorded in the Pebble manifest, and a standalone `pebble sstable check` does not currently reproduce the checksum error.
## Environment
Kubo version at the time of the incident:
```text
Kubo version: 0.41.0-d719fb87b
Repo version: 18
System version: amd64/linux
Golang version: go1.26.2
```
The machine has since been upgraded to Kubo 0.43.0, but journal logs confirm that Kubo 0.41.0 was running on August 8 when the affected SST was written.
Pebble CLI used for the investigation:
```text
github.com/cockroachdb/pebble/v2/cmd/pebble@v2.1.6
```
Datastore spec from the preserved repository:
```json
{
"path": "pebbleds",
"type": "pebbleds"
}
```
Kubo Datastore config:
```json
{
"BlockKeyCacheSize": null,
"BloomFilterSize": 0,
"GCPeriod": "1h",
"HashOnRead": false,
"Spec": {
"path": "pebbleds",
"type": "pebbleds"
},
"StorageGCWatermark": 90,
"StorageMax": "800GB"
}
```
`disableWAL` was not explicitly configured.
Filesystem/storage:
- Linux
- ext4
- software RAID1 (`md3`)
- 2 NVMe devices
- RAID currently reports `[UU]`
- current `mismatch_cnt` is 0
Memory:
```text
Error Correction Type: None
Total Width: 64 bits
Data Width: 64 bits
```
So this system does not have ECC RAM.
## Affected SST
The affected file is:
```text
67232266.sst
```
Filesystem information:
```text
size: 33744727 bytes
mtime: 2026-08-08 00:29:50.195105098 -0400
```
The Pebble manifest contains the table at L6 with the following key range:
```text
[/blocks/DYQEVUQVYQAN5BIKGEKRFZWHKQF5XNLPD36JR5QXA5FSBT3PFSZUP3I#0,SET
-
/blocks/DYQEVXHG2FFTP4GOAX44GXNDQRY5C3BQAX4UMBCGSAK6MLIHQL7KOJA#0,SET]
```
and records:
```text
size:33744727
2026-08-08T04:29:50Z
```
The manifest timestamp and filesystem mtime represent the same moment (`00:29:50 EDT == 04:29:50 UTC`).
Most importantly, the manifest records:
```text
33744727 bytes
```
and the physical SST is exactly:
```text
33744727 bytes
```
so the file does not appear to have been truncated.
## Pebble CLI diagnostics
### `pebble db check`
Run against the preserved repository:
```bash
pebble db check /pebbleds
```
Result:
```text
Found 1 WALs
- 67498829: {(/pebbleds,000)}
[JOB 1] WAL 67498829: {(/pebbleds,000)} stopped reading at offset:
(/pebbleds/67498829.log: 0); replayed 0 keys in 0 batches
checked 35462601 points and 0 tombstone
```
No additional corruption was reported while checking ~35.4M points.
### `pebble sstable check`
Run directly against the affected SST:
```bash
pebble sstable check /pebbleds/67232266.sst
```
This did not reproduce the CRC32C error. It completed without reporting a checksum failure.
### Manifest inspection
I also ran:
```bash
pebble manifest dump /pebbleds/MANIFEST-* | grep -C 15 67232266
```
This confirmed that `67232266` was an L6 SST, along with its key range, creation timestamp and recorded size.
## System events around SST creation
I inspected both the full system journal and kernel journal around:
```text
2026-08-08 00:29:50 -0400
```
including approximately ±10 minutes around the SST creation.
I did not find evidence in that window of:
- OOM kill
- host reboot/shutdown
- NVMe error
- I/O error
- ext4 error
- RAID failure
- kernel panic
- Kubo service restart
The kernel log in that window mainly contains normal Docker bridge/veth lifecycle events.
There was active workload on the host at the time, including short-lived Docker containers from our ETL infrastructure.
## Difference from #5726
I saw #5726, but this incident appears to have a different signature.
In this case:
- the SST is not zero-length
- its current size exactly matches the size recorded in the manifest
- the CRC mismatch observed through Kubo/Pebble was not reported as a computed CRC of 0
- standalone `sstable check` currently passes
## Questions
1. Is it expected that a checksum mismatch encountered while iterating an SST would later not be reproduced by `pebble sstable check` on the same preserved SST?
2. Given that the physical SST size exactly matches the manifest size, does this effectively rule out simple file truncation?
3. Could this behavior be explained by corruption outside the SST itself, caching/read behavior, memory corruption, or another Pebble structure?
4. Are there any additional Pebble commands or diagnostics worth running on the preserved 201 GB repository before I remove it?
5. Is there anything else from the original repository that would be useful to preserve or provide?
The complete original repository is currently preserved, so I can run additional read-only diagnostics if useful.
Jira issue: PEBBLE-1472
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the preserved repository and reproduce the reported `pebble db check`, `pebble sstable check`, and `pebble manifest dump` diagnostics for `67232266.sst`. Compare the iteration-time checksum failure with the standalone check and inspect the supplied WAL, manifest entry, file size, and system-event window. Done means explaining whether the mismatch is reproducible and identifying any additional read-only diagnostic that distinguishes the possible causes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100