S3 glob fails with stale HTTP 503 error even though http_retries succeeded (AWSListObjectV2::Request)
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
### What happens
A single transient S3 503 (SlowDown) on a ListObjectsV2 call fails the whole query, even when `http_retries` is configured and the retry succeeds. The query errors with:
```
HTTP Error: HTTP GET error reading 's3:///' in region 'us-east-1' (HTTP 503 Service Unavailable)
SlowDown: Please reduce your request rate.
```
This affects every `parquet_scan`/`read_parquet` over an S3 prefix (glob). Object-level GET/HEAD retries behave as configured — only the LIST path is affected. We hit this in production nightly batch jobs, where queries were failing ~100ms after starting despite `http_retries=10` (a full retry cycle at our settings would take minutes, so no retrying was visibly happening).
### Analysis
In `AWSListObjectV2::Request` (src/s3fs.cpp), the response handler records any >=400 status into an `ErrorData` captured by reference, and the code throws it after `http_util.Request()` returns. The handler runs once per attempt of the retry loop in `HTTPUtil::Request`, but nothing resets that state between attempts — so the retry machinery works, gets a 200, and the stale error from the failed first attempt is thrown anyway. The response body stringstream accumulates across attempts the same way.
### To reproduce
Self-contained, no AWS needed — a local server returns 503 for the first LIST request and recovers:
```python
import http.server, threading, duckdb
LIST_OK = b'bp/0false'
SLOWDOWN = b'SlowDownPlease reduce your request rate.'
state = {'n': 0}
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
state['n'] += 1
body, status = (SLOWDOWN, 503) if state['n'] == 1 else (LIST_OK, 200)
self.send_response(status)
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *a): pass
server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
con = duckdb.connect()
for s in [f"SET s3_endpoint='127.0.0.1:{server.server_address[1]}'", "SET s3_use_ssl=false",
"SET s3_url_style='path'", "SET s3_access_key_id='x'", "SET s3_secret_access_key='y'",
"SET s3_region='us-east-1'", "SET http_retries=10", "SET http_retry_wait_ms=300"]:
con.execute(s)
con.execute("SELECT * FROM parquet_scan('s3://b/p/*.parquet')")
```
Expected: the retry recovers and the glob proceeds (here: "no files found" for the empty listing). Actual: `HTTPException ... (HTTP 503 Service Unavailable)` after 2 requests — the retry ran, succeeded, and its result was discarded. If the server returns 503 on every request instead, retries are exhausted and the failure is correct — the bug is specific to the recovered case.
### Environment
DuckDB 1.5.4 (python), httpfs `c3f215a` (current published build); code path unchanged on `main` as of this report. macOS arm64 and Linux x64.
### Fix
Proposed in #360: reset the captured error and body at the top of the response handler so each retry attempt starts clean. Verified with an A/B build; a persistent-503 control confirms exhausted retries still fail with the detailed S3 error.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/s3fs.cpp at AWSListObjectV2::Request, then trace the retry behavior through HTTPUtil::Request. Run the self-contained local-server reproduction with http_retries enabled. Done means a recovered 503 allows the S3 glob to proceed, while persistent 503 responses still fail after retries with the detailed error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100