allow forcing full (non-range) HTTP downloads for servers with non-compliant Range/206 responses with SET force_download = true
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
Some public APIs, e.g. France's [hubeau.eaufrance.fr](https://hubeau.eaufrance.fr/) (data.gouv.fr), return non-RFC7233-compliant Range responses (206 status with mismatched Content-Length/Content-Range). httpfs treats this as fatal instead of falling back to a plain download.
### To Reproduce
```
FROM read_json('https://hubeau.eaufrance.fr/api/v1/qualite_eau_potable/resultats_dis?code_commune=31555');
Error: HTTP Error: Full download failed to URL "...": 206 (Partial Content)
```
Same root cause was already reported 2+ years ago in [#11221](https://github.com/duckdb/duckdb/issues/11221), closed as stale with no fix or workaround shipped. Still reproduces today. I've contacted Hubeau directly; a header-compliance fix on their side isn't realistic on any useful timeframe (public-administration API, limited bandwidth).
This isn't one quirky server: hubeau.eaufrance.fr is one of France's most-used open-data platforms, covering water data (drinking water quality, river quality, groundwater levels, hydrometry, etc.).
### Proposed solution
A session-level setting to force a full sequential GET, skipping HEAD/Range logic and Content-Length/Content-Range validation, with the common force_download setting, so have it work with this setting (it doesn't currently):
```
SET force_download = true;
FROM read_json('https://hubeau.eaufrance.fr/api/v1/qualite_eau_potable/resultats_dis?code_commune=31555');
```
### Alternatives considered
Pre-downloading with curl or Python (requests + temp file) works, but defeats the point of httpfs and isn't viable in SQL-only contexts (CLI, dbt, BI tools embedding DuckDB).
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no implementation files or tests. Start by tracing the force_download setting and the read_json HTTP path in httpfs, then inspect handling of HEAD, Range, and 206 responses; done means the Hubeau reproduction works with SET force_download = true without the current validation failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100