S3 region redirect causes self-deadlock in file initialization and silent data corruption in cache
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
### Description
When an S3 request is sent to the wrong regional endpoint, S3 returns HTTP 301 with an `x-amz-bucket-region` header indicating the correct region. The httpfs extension's redirect handling has two bugs that cause queries to hang indefinitely or fail with corrupt data.
### Bug 1: Self-deadlock in `S3FileHandle::Initialize()`
When S3 returns a 301 during file initialization, the retry path calls `HTTPFileHandle::Initialize()` again on the same thread. This re-acquires `CachedFile::lock`, which is a non-recursive `std::mutex`, causing an immediate self-deadlock. The query hangs indefinitely and cannot be cancelled.
**Root cause:** `CachedFile::lock` is `std::mutex` (non-recursive), and the redirect retry re-enters the lock on the same thread.
**Additionally:** The failed first attempt stores the S3 error/redirect XML body into the file cache as if it were valid file content. Even if the deadlock is resolved, subsequent reads return the XML error response instead of actual file data, producing errors like "Incorrect Avro container file magic number."
### Bug 2: No redirect handling in `S3FileSystem::GetRangeRequest()`
Range GET requests (used for reading data file segments, e.g., during JOINs across multiple Iceberg tables) have no 301 handling. When S3 returns a redirect, it is thrown as an unrecoverable `IOException` rather than retrying with the corrected region.
This breaks queries that JOIN multiple tables where data files (not just metadata) reside in a different region than the configured endpoint.
### Reproduction
1. Create an Iceberg table with S3 data in `us-west-2`
2. Configure the DuckDB S3 region as `us-east-1`
3. Query the table → **hangs indefinitely** (Bug 1)
4. If Bug 1 is worked around (e.g., using `recursive_mutex`), reading Iceberg manifests fails with "Incorrect Avro container file magic number" because the cache holds stale XML error data
5. JOIN two Iceberg tables where data files trigger cross-region redirects → **IOException** (Bug 2)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing S3FileHandle::Initialize(), CachedFile::lock, and S3FileSystem::GetRangeRequest() through the redirect reproduction. Verify that a corrected-region retry does not re-enter the non-recursive lock or cache the XML error body, and that range GET redirects retry successfully without corrupting reads or raising an unrecoverable IOException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100