HTTPFS fully downloads a file when HEAD returns 200 without Content-Length, even though Range requests are supported
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
### What happens?
## Description
When opening a remote Parquet file over HTTP, `httpfs` performs a full download if the server returns a successful `HEAD 200` response without either `Content-Length` or `Content-Range`.
This happens even when the server correctly supports byte-range requests and returns `206 Partial Content` for ranged GET requests.
It also happens with:
```sql
SET auto_fallback_to_full_download = false;
```
That setting does not affect this code path because no Range request is attempted before `FullDownload()` is called.
## Environment
```text
DuckDB v1.5.4
Linux amd64
httpfs
```
## Reproduction URL
Word of warning : The missing headers for this URL might be fixed soon (a patch is in qualification on our side). However the full description below gives all the relevant information.
```text
https://data.geopf.fr/chunk/telechargement/download/BDTOPO_PQT/BDTOPO_3-5_TOUSTHEMES_GEOPARQUET_WGS84G_FRA_2026-03-15/equipement_de_transport.parquet
```
The file size is:
```text
34223378 bytes
```
## Server behaviour
The `HEAD` request succeeds but does not provide the file size:
```bash
URL='https://data.geopf.fr/chunk/telechargement/download/BDTOPO_PQT/BDTOPO_3-5_TOUSTHEMES_GEOPARQUET_WGS84G_FRA_2026-03-15/equipement_de_transport.parquet'
curl -I "$URL"
```
Relevant response:
```http
HTTP/2 200
```
The response does not contain:
```http
Content-Length
Content-Range
Accept-Ranges
```
However, the same server correctly supports byte-range requests:
```bash
curl \
--range 0-1 \
--dump-header - \
--output /dev/null \
"$URL"
```
Relevant response:
```http
HTTP/2 206
Content-Length: 2
Content-Range: bytes 0-1/34223378
```
## DuckDB reproduction
```sql
INSTALL httpfs;
LOAD httpfs;
SET auto_fallback_to_full_download = false;
SELECT count(*)
FROM read_parquet(
'https://data.geopf.fr/chunk/telechargement/download/BDTOPO_PQT/BDTOPO_3-5_TOUSTHEMES_GEOPARQUET_WGS84G_FRA_2026-03-15/equipement_de_transport.parquet'
);
```
With HTTP logging enabled, the request sequence is:
```text
HEAD
→ HTTP 200
→ no Content-Length or Content-Range
GET
→ no Range request header
→ HTTP 200
→ Content-Length: 34223378
→ full file download
```
## Expected behaviour
When a successful `HEAD` response does not provide a usable file size, `httpfs` should try a small ranged GET before deciding to download the complete file.
For example:
```http
Range: bytes=0-1
```
The total file size can then be obtained from:
```http
Content-Range: bytes 0-1/34223378
```
If the Range request is genuinely unsupported, the existing fallback or error behaviour can then be applied according to `auto_fallback_to_full_download`.
## Actual behaviour
The successful but incomplete `HEAD` response leaves `length` equal to zero.
`HTTPFileHandle::Initialize()` then treats `length == 0` as a reason to call `FullDownload()`.
Consequently, the whole remote Parquet file is downloaded even though the server supports efficient ranged access.
## Relevant code
`HTTPFileHandle::LoadFileInfo()` only tries a Range request when the HEAD response status is not `200`:
https://github.com/duckdb/duckdb-httpfs/blob/df92a34d29eb589591adfadba89fa8df874e54ea/src/httpfs.cpp#L1013-L1042
After a successful HEAD request, the file size is obtained only from `Content-Range` or `Content-Length`:
https://github.com/duckdb/duckdb-httpfs/blob/df92a34d29eb589591adfadba89fa8df874e54ea/src/httpfs.cpp#L1043-L1051
If neither header exists, `length` remains zero.
That subsequently triggers a full download:
https://github.com/duckdb/duckdb-httpfs/blob/df92a34d29eb589591adfadba89fa8df874e54ea/src/httpfs.cpp#L1158-L1165
The `auto_fallback_to_full_download` setting is checked in the Range-request failure path:
https://github.com/duckdb/duckdb-httpfs/blob/df92a34d29eb589591adfadba89fa8df874e54ea/src/httpfs.cpp#L652-L667
That path is never reached in this case because no Range request is attempted.
## Comparison with GDAL
GDAL `/vsicurl/` handles this situation more defensively.
When a `HEAD 200` response does not provide the file size, GDAL retries with a GET to retrieve the response headers, interrupts the body transfer, and then performs the actual random reads using Range requests.
Relevant GDAL code:
https://github.com/OSGeo/gdal/blob/de2440fd7aa21d983e982a1f6b4ff3487a97769c/port/cpl_vsil_curl.cpp#L1565-L1606
## Suggested fix
In `HTTPFileHandle::LoadFileInfo()`, when all of the following conditions are true:
```text
HEAD status == 200
Content-Length is absent
Content-Range is absent
file is opened for reading
```
try a small ranged GET before performing a full download.
Conceptually:
```cpp
if (head_response_is_200 &&
!content_size.IsValid() &&
flags.OpenForReading()) {
auto range_res =
hfs.GetRangeRequest(*this, path, {}, 0, nullptr, 2);
// Parse the total file size from Content-Range.
}
```
This would preserve efficient remote Parquet access for servers whose HEAD responses omit representation metadata, while retaining the existing fallback behaviour when Range requests are actually unsupported.
### To Reproduce
```sql
INSTALL httpfs;
LOAD httpfs;
SET auto_fallback_to_full_download = false;
SELECT count(*)
FROM read_parquet(
'https://data.geopf.fr/chunk/telechargement/download/BDTOPO_PQT/BDTOPO_3-5_TOUSTHEMES_GEOPARQUET_WGS84G_FRA_2026-03-15/equipement_de_transport.parquet'
);
```
### OS:
Ubuntu 24.04
### DuckDB Version:
v1.5.4 (Variegata) 08e34c447b
### DuckDB Client:
duckdb command line
### Hardware:
amd64
### Full Name:
Emmanuel Séguin
### Affiliation:
IGN France
### Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?
- [x] Yes, I have
### Did you include all code required to reproduce the issue?
- [x] Yes, I have
### Did you include all relevant data sets for reproducing the issue?
Yes
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/httpfs.cpp at HTTPFileHandle::LoadFileInfo() and HTTPFileHandle::Initialize(), then run the provided DuckDB SQL reproduction with HTTP logging enabled. Trace the HEAD, Range-request failure, and FullDownload paths, and verify that an incomplete successful HEAD causes a small ranged request before any full download while unsupported ranges retain the existing fallback behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100