duckdb / duckdb/duckdb-httpfs

allow forcing full (non-range) HTTP downloads for servers with non-compliant Range/206 responses with SET force_download = true

Open
#407 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

Some public APIs, e.g. France's [hubeau.eaufrance.fr](https://hubeau.eaufrance.fr/) (data.gouv.fr), return non-RFC7233-compliant Range responses (206 status with mismatched Content-Length/Content-Range). httpfs treats this as fatal instead of falling back to a plain download.

### To Reproduce
```
FROM read_json('https://hubeau.eaufrance.fr/api/v1/qualite_eau_potable/resultats_dis?code_commune=31555');
Error: HTTP Error: Full download failed to URL "...": 206 (Partial Content)
```
Same root cause was already reported 2+ years ago in [#11221](https://github.com/duckdb/duckdb/issues/11221), closed as stale with no fix or workaround shipped. Still reproduces today. I've contacted Hubeau directly; a header-compliance fix on their side isn't realistic on any useful timeframe (public-administration API, limited bandwidth).

This isn't one quirky server: hubeau.eaufrance.fr is one of France's most-used open-data platforms, covering water data (drinking water quality, river quality, groundwater levels, hydrometry, etc.).

### Proposed solution
A session-level setting to force a full sequential GET, skipping HEAD/Range logic and Content-Length/Content-Range validation, with the common force_download setting, so have it work with this setting (it doesn't currently):

```
SET force_download = true;
FROM read_json('https://hubeau.eaufrance.fr/api/v1/qualite_eau_potable/resultats_dis?code_commune=31555');
```
### Alternatives considered
Pre-downloading with curl or Python (requests + temp file) works, but defeats the point of httpfs and isn't viable in SQL-only contexts (CLI, dbt, BI tools embedding DuckDB).

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no implementation files or tests. Start by tracing the force_download setting and the read_json HTTP path in httpfs, then inspect handling of HEAD, Range, and 206 responses; done means the Hubeau reproduction works with SET force_download = true without the current validation failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.