duckdb / duckdb/duckdb-httpfs

ETag-matched cache entry can carry wrong file size after concurrent write, causing permanent "No magic bytes found at end of file

Open
#338 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

## What happens
When a remote parquet file is replaced (e.g. via S3 PutObject) while a concurrent DuckDB read is in progress, the httpfs metadata cache can store a {new_ETag, wrong_length} entry. All subsequent reads on the same connection then seek to the wrong offset and fail permanently with:

```No magic bytes found at end of file```

The error persists for the lifetime of the connection. The only recovery is recreating the `DuckdbClient`. The root cause might be that `ExternalFileCache::IsValid()` validates entries by ETag comparison only:

```c++
if (!current_version_tag.empty() || !cached_version_tag.empty()) {
return cached_version_tag == current_version_tag; // length is never checked
}
```

Once the new ETag is cached alongside the wrong length, every subsequent IsValid() returns true and the wrong length is used indefinitely.

## To reproduce
It is difficult to reproduce, but below is how it was triggered in our pipeline.
1. Upload a parquet file to any S3-compatible store
2. Start a DuckDB read_parquet(...) query against it
3. Concurrently replace the file with a differently-sized version (e.g. via aws s3 cp or rclone)
4. The current query or the next one will fail with "No magic bytes" and continue failing on every subsequent query

## Expected behaviour
Length mismatch from the same E-tag can be detected and the error self-heals after the remote parquet is completely replaced.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at ExternalFileCache::IsValid() and trace the httpfs metadata-cache path used by read_parquet(...), then reproduce the concurrent S3 replacement if possible. Done means a same-ETag length mismatch is detected and subsequent reads recover after the remote file is replaced, with regression coverage for the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.