Would it be possible to return empty rows for failing requests?
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
Hey,
I find myself writing often something like this:
```sql
FROM read_json([
'http://example.com/data.json',
'http://non-existent.test/data.json'
]);
```
or
```
FROM read_json(getvariable('resource_urls'));
```
or
```
FROM read_text(getvariable('resource_urls'));
```
If there are hundreds of URLs every now and then some of the resources will returns errors like these:
```
Could not establish connection error for HTTP HEAD to 'http://non-existent.test/data.json'
Unable to connect to URL "http://example.com/data.json": 404 (Not Found).
```
In this situation all other results will go to waste if 1 url from 300 urls will fail.
There are already excellent configs I can use to increase the retry count and wait times:
```
SET http_retries = 10;
SET http_retry_wait_ms = 2000;
```
Sometimes it's not enough to retry and the resource is not there for a hour or a day or for a week or for a month but will eventually appear again.
Would there be a way to allow missing or failing resource in the larger arrays and still be able to load all other files?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failing multi-URL queries using read_json and read_text, including the resource_urls variable form, and review how HTTP errors currently affect the larger input array. Done should mean unavailable resources no longer discard successful results, while the requested empty-row behavior is defined for connection failures and HTTP 404 responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data, networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100