duckdb / duckdb/duckdb-httpfs

Long-running parallel parquet reads from public S3 fail on 1.5.x (work on 1.4.4)

Open
#334 15 comments 3 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

### What happens

On `duckdb==1.5.3` (`osx_arm64`, `httpfs 52afb42`), running a long-running
parquet read against AWS S3 fails partway through with one of the following
errors:

```
IO Error: SSL connect error error for HTTP GET to 'https://.s3..amazonaws.com/.../*.parquet'
IO Error: SSL connection failed error for HTTP GET to 'https://.s3..amazonaws.com/.../*.parquet'
IO Error: Timeout was reached error for HTTP GET to 'https://.s3..amazonaws.com/.../*.parquet'
HTTP Error: HTTP GET error on 'https://.s3..amazonaws.com/.../*.parquet' (HTTP 0 Internal Server Error)
```

The exact wording varies run-to-run on the same query. The first dozens of
parquet `Range`-GETs return 206 Partial Content fine; failures only occur
once the parquet reader has been streaming many parallel range GETs for a
minute or two.

The same query against the same data succeeds on `duckdb==1.4.4`.

We originally hit this on an authenticated workload against an Iceberg REST
catalog whose data files live in S3, but it reproduces just as cleanly on a
public bucket with no credentials, so credentials/Iceberg are not part of
the problem.

### Versions where reproduced

| duckdb | httpfs | result |
| ------ | -------------------------------------------------------- | ----------------------------------- |
| 1.4.4 | (default `httplib`) | ✅ succeeds, ~9 min full read |
| 1.5.3 | 52afb42 (default `curl`) | ❌ fails @ ~80–160 s (varies) |
| 1.5.3 | 52afb42 + `SET httpfs_client_implementation = 'httplib'` | ❌ fails @ ~150 s (delayed, varies) |

So the issue is **not** specific to the new curl-based HTTP backend introduced
in [1.5.0](https://duckdb.org/2026/03/09/announcing-duckdb-150.html#network-stack):
both backends fail in 1.5.3, just at different latencies. `httplib` only delays
the failure threshold; for queries that take a few minutes it still fails.

Network is fine: `curl -v https://overturemaps-us-west-2.s3.us-west-2.amazonaws.com/`
from the same host completes a TLS handshake cleanly with the system CA store,
and 1.4.4 on the same host streams hundreds of 206 Partial Content responses
without issue for the same query.

### Environment

- Host: macOS 26.5 arm64 (Apple Silicon)
- Python 3.12
- `duckdb==1.5.3` Python wheel (installed via `uvx`)
- `httpfs` extension version `52afb42`
- Public AWS S3 bucket `overturemaps-us-west-2` (single DNS label,
no dots → not the `duckdb-httpfs#283` bucket-with-dots regression)
- No corporate proxy, no TLS interception

### Minimal reproducer (no credentials required)

```python
"""Reproducer using the public Overture Maps S3 release."""
import os, tempfile, time, duckdb

con = duckdb.connect()
con.sql("INSTALL httpfs"); con.sql("LOAD httpfs")

out = os.path.join(tempfile.gettempdir(), "overture_ocean.parquet")
t0 = time.time()
try:
con.execute(f"""
COPY (
SELECT geometry
FROM read_parquet(
's3://overturemaps-us-west-2/release/2026-06-17.0/theme=base/type=water/*',
filename = true,
hive_partitioning = 1
)
WHERE subtype = 'ocean'
) TO '{out}' (FORMAT PARQUET)
""")
print(f"OK in {time.time()-t0:.1f}s")
except Exception as e:
print(f"FAILED in {time.time()-t0:.1f}s: {type(e).__name__}: {e}")
```

The bucket is public, so no S3 secret or AWS credentials are required. If
your shell has `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` exported, add an
unsigned secret first to force anonymous access:

```sql
CREATE OR REPLACE SECRET overture_public (
TYPE S3, PROVIDER config, REGION 'us-west-2', KEY_ID '', SECRET ''
);
```

Run twice for comparison:

```bash
uvx --with duckdb==1.4.4 python repro.py # succeeds, ~9 min full read
uvx --with duckdb==1.5.3 python repro.py # fails partway through
```

The Overture release path (`release/2026-05-20.0/...`) is just the most
recent release at the time of filing; any release with a similar volume of
parquet partitions reproduces.

### Sample HTTP debug log on the failing run (1.5.3)

The first dozens of parquet `Range` GETs return 206 Partial Content fine.
After ~30–60 s of streaming, the next batch of GETs times out and DuckDB
retries 3 times before surfacing one of the error messages above. With
`SET enable_logging = true; SET logging_level = 'debug'; SET logging_storage = 'stdout';`:

```
... HTTP DEBUG {request: HEAD ..., response: PartialContent_206, ...}
... HTTP DEBUG {request: GET ..., headers: {Range='bytes=...'}, response: PartialContent_206, ...}
... (many successful 206 responses)
... HTTP DEBUG {request: GET ..., response: NULL} # transport-level failure
... HTTP DEBUG {request: GET ..., response: NULL} # retry 1
... HTTP DEBUG {request: GET ..., response: NULL} # retry 2
... HTTP DEBUG {request: GET ..., response: NULL} # retry 3
... Transaction Rollback
FAILED: IO Error: SSL connect error error for HTTP GET to 'https://...'
```

### Settings tried

None of the following resolve the issue on 1.5.3:

- `SET httpfs_client_implementation = 'httplib'`
- `SET http_timeout = 300000`
- `SET http_retries = 10`
- `SET http_retry_wait_ms = 500`
- `SET http_keep_alive = false`
- `SET enable_curl_server_cert_verification = false`
- `SET httpfs_connection_caching = true`
- `SET threads = 1` (didn't try yet, will try if requested)

`FORCE INSTALL httpfs` + clearing `~/.duckdb/extensions` and re-downloading
gives the same `52afb42` extension and the same failure.

### Hypothesis

Both HTTP backends in 1.5.3 fail on long-running parallel S3 fetches initiated
by the parquet reader, while the same workload works on 1.4.4. Since
this affects both the curl and httplib paths in 1.5.3, the regression seems to
be in something _above_ the HTTP client — possibly how httpfs in 1.5.3
hands connections out to many parallel parquet readers, or how
retry/timeout state is shared across them. The connection-caching changes in
1.5 (`httpfs_connection_caching` setting was added) seem like a plausible
culprit, but flipping it on/off doesn't help.

### Related issues

- [duckdb/duckdb#19107](https://github.com/duckdb/duckdb/issues/19107) — Glue
catalog SSL curl errors; appears to share the same family of symptoms but
was reported during catalog attach, not during parquet read.
- [duckdb/duckdb-python#398](https://github.com/duckdb/duckdb-python/issues/398) —
SSL peer certificate errors in v1.5; root-caused there to bucket-name dots,
which is **not** our case (bucket name is a single DNS label).
- [duckdb/duckdb-httpfs#283](https://github.com/duckdb/duckdb-httpfs/issues/283) —
Bucket-name dots regression; not our case.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running repro.py with DuckDB 1.4.4 and 1.5.3 against the public S3 path, then compare the curl and httplib settings and the HTTP debug logs. Read the httpfs connection-caching, retry, and parallel range-read entry points implicated by the report. Done means the long-running read completes reliably on 1.5.3 without transport failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
databases, networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.