duckdb / duckdb/duckdb-httpfs

HTTPFS fully downloads a file when HEAD returns 200 without Content-Length, even though Range requests are supported

Open
#401 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

### What happens?

## Description

When opening a remote Parquet file over HTTP, `httpfs` performs a full download if the server returns a successful `HEAD 200` response without either `Content-Length` or `Content-Range`.

This happens even when the server correctly supports byte-range requests and returns `206 Partial Content` for ranged GET requests.

It also happens with:

```sql
SET auto_fallback_to_full_download = false;
```

That setting does not affect this code path because no Range request is attempted before `FullDownload()` is called.

## Environment

```text
DuckDB v1.5.4
Linux amd64
httpfs
```

## Reproduction URL

Word of warning : The missing headers for this URL might be fixed soon (a patch is in qualification on our side). However the full description below gives all the relevant information.

```text
https://data.geopf.fr/chunk/telechargement/download/BDTOPO_PQT/BDTOPO_3-5_TOUSTHEMES_GEOPARQUET_WGS84G_FRA_2026-03-15/equipement_de_transport.parquet
```

The file size is:

```text
34223378 bytes
```

## Server behaviour

The `HEAD` request succeeds but does not provide the file size:

```bash
URL='https://data.geopf.fr/chunk/telechargement/download/BDTOPO_PQT/BDTOPO_3-5_TOUSTHEMES_GEOPARQUET_WGS84G_FRA_2026-03-15/equipement_de_transport.parquet'

curl -I "$URL"
```

Relevant response:

```http
HTTP/2 200
```

The response does not contain:

```http
Content-Length
Content-Range
Accept-Ranges
```

However, the same server correctly supports byte-range requests:

```bash
curl \
--range 0-1 \
--dump-header - \
--output /dev/null \
"$URL"
```

Relevant response:

```http
HTTP/2 206
Content-Length: 2
Content-Range: bytes 0-1/34223378
```

## DuckDB reproduction

```sql
INSTALL httpfs;
LOAD httpfs;

SET auto_fallback_to_full_download = false;

SELECT count(*)
FROM read_parquet(
'https://data.geopf.fr/chunk/telechargement/download/BDTOPO_PQT/BDTOPO_3-5_TOUSTHEMES_GEOPARQUET_WGS84G_FRA_2026-03-15/equipement_de_transport.parquet'
);
```

With HTTP logging enabled, the request sequence is:

```text
HEAD
→ HTTP 200
→ no Content-Length or Content-Range

GET
→ no Range request header
→ HTTP 200
→ Content-Length: 34223378
→ full file download
```

## Expected behaviour

When a successful `HEAD` response does not provide a usable file size, `httpfs` should try a small ranged GET before deciding to download the complete file.

For example:

```http
Range: bytes=0-1
```

The total file size can then be obtained from:

```http
Content-Range: bytes 0-1/34223378
```

If the Range request is genuinely unsupported, the existing fallback or error behaviour can then be applied according to `auto_fallback_to_full_download`.

## Actual behaviour

The successful but incomplete `HEAD` response leaves `length` equal to zero.

`HTTPFileHandle::Initialize()` then treats `length == 0` as a reason to call `FullDownload()`.

Consequently, the whole remote Parquet file is downloaded even though the server supports efficient ranged access.

## Relevant code

`HTTPFileHandle::LoadFileInfo()` only tries a Range request when the HEAD response status is not `200`:

https://github.com/duckdb/duckdb-httpfs/blob/df92a34d29eb589591adfadba89fa8df874e54ea/src/httpfs.cpp#L1013-L1042

After a successful HEAD request, the file size is obtained only from `Content-Range` or `Content-Length`:

https://github.com/duckdb/duckdb-httpfs/blob/df92a34d29eb589591adfadba89fa8df874e54ea/src/httpfs.cpp#L1043-L1051

If neither header exists, `length` remains zero.

That subsequently triggers a full download:

https://github.com/duckdb/duckdb-httpfs/blob/df92a34d29eb589591adfadba89fa8df874e54ea/src/httpfs.cpp#L1158-L1165

The `auto_fallback_to_full_download` setting is checked in the Range-request failure path:

https://github.com/duckdb/duckdb-httpfs/blob/df92a34d29eb589591adfadba89fa8df874e54ea/src/httpfs.cpp#L652-L667

That path is never reached in this case because no Range request is attempted.

## Comparison with GDAL

GDAL `/vsicurl/` handles this situation more defensively.

When a `HEAD 200` response does not provide the file size, GDAL retries with a GET to retrieve the response headers, interrupts the body transfer, and then performs the actual random reads using Range requests.

Relevant GDAL code:

https://github.com/OSGeo/gdal/blob/de2440fd7aa21d983e982a1f6b4ff3487a97769c/port/cpl_vsil_curl.cpp#L1565-L1606

## Suggested fix

In `HTTPFileHandle::LoadFileInfo()`, when all of the following conditions are true:

```text
HEAD status == 200
Content-Length is absent
Content-Range is absent
file is opened for reading
```

try a small ranged GET before performing a full download.

Conceptually:

```cpp
if (head_response_is_200 &&
!content_size.IsValid() &&
flags.OpenForReading()) {
auto range_res =
hfs.GetRangeRequest(*this, path, {}, 0, nullptr, 2);

// Parse the total file size from Content-Range.
}
```

This would preserve efficient remote Parquet access for servers whose HEAD responses omit representation metadata, while retaining the existing fallback behaviour when Range requests are actually unsupported.

### To Reproduce

```sql
INSTALL httpfs;
LOAD httpfs;

SET auto_fallback_to_full_download = false;

SELECT count(*)
FROM read_parquet(
'https://data.geopf.fr/chunk/telechargement/download/BDTOPO_PQT/BDTOPO_3-5_TOUSTHEMES_GEOPARQUET_WGS84G_FRA_2026-03-15/equipement_de_transport.parquet'
);
```

### OS:

Ubuntu 24.04

### DuckDB Version:

v1.5.4 (Variegata) 08e34c447b

### DuckDB Client:

duckdb command line

### Hardware:

amd64

### Full Name:

Emmanuel Séguin

### Affiliation:

IGN France

### Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

- [x] Yes, I have

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant data sets for reproducing the issue?

Yes

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/httpfs.cpp at HTTPFileHandle::LoadFileInfo() and HTTPFileHandle::Initialize(), then run the provided DuckDB SQL reproduction with HTTP logging enabled. Trace the HEAD, Range-request failure, and FullDownload paths, and verify that an incomplete successful HEAD causes a small ranged request before any full download while unsupported ranges retain the existing fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.