Loki stops returning logs when meeting 'invalid chunk checksum'
- Dominant language
- Go
- Stars
- 28.9k
- Forks
- 4.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 395
Description
**Describe the bug**
When Loki has invalid / corrupted / empty chunks, 'invalid chunk checksum' is returned even if other chunks are otherwise valid.
**To Reproduce**
This bug was met on a machine which filled the disk and the filled the memory. This caused some of the chunks to get corrupted, which is perfectly expected in this case. After restarting the machine, logs spanning invalid chunks could not longer be retrieved, even if a subset of the chunks was still valid.
This can be easily reproduced by:
1. Stopping a running instance of loki
2. Truncating an arbitrary chunk to be empty
3. Restarting Loki
4. Querying for logs spanning the deleted chunk
**Expected behavior**
If possible, logs stored in valid chunks should still be returned as they may contain valuable information to understand how the faulty machine ended up filling the disk and the memory for example.
**Environment:**
- Infrastructure: bare-metal
- Deployment tool: single instance mode, launched via systemd
**What we tried:**
Blindly deleting the 0-length chunks manually triggered a "no such file or directory".
Empirically moving the request start date by 17s allowed Loki to return the logs. However, I'm not sure how to correctly determine the time range to exclude.
Patching `GetParallelChunks` to never return the last error if chunks could be retrieved allowed Loki to return all logs from valid chunks. However, the information that there are invalid logs is no longer propagated nor logged (like in fetchLazyChunks).
```patch
diff --git a/pkg/storage/chunk/client/util/parallel_chunk_fetch.go b/pkg/storage/chunk/client/util/parallel_chunk_fetch.go
index 446644849..6e5f64ea2 100644
--- a/pkg/storage/chunk/client/util/parallel_chunk_fetch.go
+++ b/pkg/storage/chunk/client/util/parallel_chunk_fetch.go
@@ -70,7 +70,7 @@ func GetParallelChunks(ctx context.Context, maxParallel int, chunks []chunk.Chun
}
// Return any chunks we did receive: a partial result may be useful
- return result, lastErr
+ return result, nil
}
func min(a, b int) int {
```
Contributor guide
Assessment
This issue has not been assessed yet.