duckdb / duckdb/duckdb-httpfs

S3 region redirect causes self-deadlock in file initialization and silent data corruption in cache

Open
#347 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

### Description

When an S3 request is sent to the wrong regional endpoint, S3 returns HTTP 301 with an `x-amz-bucket-region` header indicating the correct region. The httpfs extension's redirect handling has two bugs that cause queries to hang indefinitely or fail with corrupt data.

### Bug 1: Self-deadlock in `S3FileHandle::Initialize()`

When S3 returns a 301 during file initialization, the retry path calls `HTTPFileHandle::Initialize()` again on the same thread. This re-acquires `CachedFile::lock`, which is a non-recursive `std::mutex`, causing an immediate self-deadlock. The query hangs indefinitely and cannot be cancelled.

**Root cause:** `CachedFile::lock` is `std::mutex` (non-recursive), and the redirect retry re-enters the lock on the same thread.

**Additionally:** The failed first attempt stores the S3 error/redirect XML body into the file cache as if it were valid file content. Even if the deadlock is resolved, subsequent reads return the XML error response instead of actual file data, producing errors like "Incorrect Avro container file magic number."

### Bug 2: No redirect handling in `S3FileSystem::GetRangeRequest()`

Range GET requests (used for reading data file segments, e.g., during JOINs across multiple Iceberg tables) have no 301 handling. When S3 returns a redirect, it is thrown as an unrecoverable `IOException` rather than retrying with the corrected region.

This breaks queries that JOIN multiple tables where data files (not just metadata) reside in a different region than the configured endpoint.

### Reproduction

1. Create an Iceberg table with S3 data in `us-west-2`
2. Configure the DuckDB S3 region as `us-east-1`
3. Query the table → **hangs indefinitely** (Bug 1)
4. If Bug 1 is worked around (e.g., using `recursive_mutex`), reading Iceberg manifests fails with "Incorrect Avro container file magic number" because the cache holds stale XML error data
5. JOIN two Iceberg tables where data files trigger cross-region redirects → **IOException** (Bug 2)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing S3FileHandle::Initialize(), CachedFile::lock, and S3FileSystem::GetRangeRequest() through the redirect reproduction. Verify that a corrected-region retry does not re-enter the non-recursive lock or cache the XML error body, and that range GET redirects retry successfully without corrupting reads or raising an unrecoverable IOException.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.