Error reading JSON API over HTTP unless force_download=true (server returns 405 on HEAD, DuckDB falls back to broken range requests)
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
### What happens?
Reading a JSON API endpoint over HTTP fails by default:
```
FROM read_json('https://opendata.adsb.fi/api/v2/lat/43.6044/lon/1.432/dist/20');
SQL Error: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: HTTP Error: HTTP GET error on 'https://opendata.adsb.fi/api/v2/lat/43.6044/lon/1.432/dist/20' (HTTP 0 Internal Server Error)
```
### To Reproduce
It works fine if I explicitly set:
`SET force_download = true;`
I'd expect DuckDB to handle this gracefully by default (e.g. detect that range requests aren't supported and fall back to a plain GET automatically), rather than requiring the user to manually discover and set force_download.
### HTTP logging detail
With `CALL enable_logging('HTTP');` enabled, I can see:
DuckDB issues a HEAD request first → server responds 405 Method Not Allowed
DuckDB then issues 4x GET requests with Range: bytes=0-1 → each response comes back as NULL
So the server (opendata.adsb.fi) doesn't support HEAD or byte-range requests, and DuckDB's httpfs client doesn't fall back to a normal full GET automatically, it just fails.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the HTTP client path used by read_json and compare the default request flow with the force_download=true flow. Reproduce the endpoint with CALL enable_logging('HTTP'); and verify that a server rejecting HEAD and byte-range requests is handled by a normal GET, with the existing successful behavior preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100