duckdb / duckdb/duckdb-httpfs

Error reading JSON API over HTTP unless force_download=true (server returns 405 on HEAD, DuckDB falls back to broken range requests)

Open
#408 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

### What happens?
Reading a JSON API endpoint over HTTP fails by default:

```
FROM read_json('https://opendata.adsb.fi/api/v2/lat/43.6044/lon/1.432/dist/20');
SQL Error: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: HTTP Error: HTTP GET error on 'https://opendata.adsb.fi/api/v2/lat/43.6044/lon/1.432/dist/20' (HTTP 0 Internal Server Error)
```

### To Reproduce
It works fine if I explicitly set:
`SET force_download = true;`

I'd expect DuckDB to handle this gracefully by default (e.g. detect that range requests aren't supported and fall back to a plain GET automatically), rather than requiring the user to manually discover and set force_download.

### HTTP logging detail

With `CALL enable_logging('HTTP');` enabled, I can see:

DuckDB issues a HEAD request first → server responds 405 Method Not Allowed
DuckDB then issues 4x GET requests with Range: bytes=0-1 → each response comes back as NULL

So the server (opendata.adsb.fi) doesn't support HEAD or byte-range requests, and DuckDB's httpfs client doesn't fall back to a normal full GET automatically, it just fails.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the HTTP client path used by read_json and compare the default request flow with the force_download=true flow. Reproduce the endpoint with CALL enable_logging('HTTP'); and verify that a server rejecting HEAD and byte-range requests is handled by a normal GET, with the existing successful behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.