duckdb / duckdb/duckdb-httpfs

S3 glob fails with stale HTTP 503 error even though http_retries succeeded (AWSListObjectV2::Request)

Open
#361 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

### What happens

A single transient S3 503 (SlowDown) on a ListObjectsV2 call fails the whole query, even when `http_retries` is configured and the retry succeeds. The query errors with:

```
HTTP Error: HTTP GET error reading 's3:///' in region 'us-east-1' (HTTP 503 Service Unavailable)

SlowDown: Please reduce your request rate.
```

This affects every `parquet_scan`/`read_parquet` over an S3 prefix (glob). Object-level GET/HEAD retries behave as configured — only the LIST path is affected. We hit this in production nightly batch jobs, where queries were failing ~100ms after starting despite `http_retries=10` (a full retry cycle at our settings would take minutes, so no retrying was visibly happening).

### Analysis

In `AWSListObjectV2::Request` (src/s3fs.cpp), the response handler records any >=400 status into an `ErrorData` captured by reference, and the code throws it after `http_util.Request()` returns. The handler runs once per attempt of the retry loop in `HTTPUtil::Request`, but nothing resets that state between attempts — so the retry machinery works, gets a 200, and the stale error from the failed first attempt is thrown anyway. The response body stringstream accumulates across attempts the same way.

### To reproduce

Self-contained, no AWS needed — a local server returns 503 for the first LIST request and recovers:

```python
import http.server, threading, duckdb

LIST_OK = b'bp/0false'
SLOWDOWN = b'SlowDownPlease reduce your request rate.'
state = {'n': 0}

class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
state['n'] += 1
body, status = (SLOWDOWN, 503) if state['n'] == 1 else (LIST_OK, 200)
self.send_response(status)
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *a): pass

server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()

con = duckdb.connect()
for s in [f"SET s3_endpoint='127.0.0.1:{server.server_address[1]}'", "SET s3_use_ssl=false",
"SET s3_url_style='path'", "SET s3_access_key_id='x'", "SET s3_secret_access_key='y'",
"SET s3_region='us-east-1'", "SET http_retries=10", "SET http_retry_wait_ms=300"]:
con.execute(s)
con.execute("SELECT * FROM parquet_scan('s3://b/p/*.parquet')")
```

Expected: the retry recovers and the glob proceeds (here: "no files found" for the empty listing). Actual: `HTTPException ... (HTTP 503 Service Unavailable)` after 2 requests — the retry ran, succeeded, and its result was discarded. If the server returns 503 on every request instead, retries are exhausted and the failure is correct — the bug is specific to the recovered case.

### Environment

DuckDB 1.5.4 (python), httpfs `c3f215a` (current published build); code path unchanged on `main` as of this report. macOS arm64 and Linux x64.

### Fix

Proposed in #360: reset the captured error and body at the top of the response handler so each retry attempt starts clean. Verified with an A/B build; a persistent-503 control confirms exhausted retries still fail with the detailed S3 error.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/s3fs.cpp at AWSListObjectV2::Request, then trace the retry behavior through HTTPUtil::Request. Run the self-contained local-server reproduction with http_retries enabled. Done means a recovered 503 allows the S3 glob to proceed, while persistent 503 responses still fail after retries with the detailed error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.