Read DataFrame from URL
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
It is a common use-case to dynamically download datasets (usually CSV or JSON) from third-party APIs, particularly when the data changes with time.
**Describe the solution you'd like**
It would be nice to be able to pass any URL to the `DataFrame::read_*` and stream the data in. Currently a confusing error is displayed if the URL doesn't correspond to a supported source like S3.
```Error: Internal error: No suitable object store found for https://<...>.csv. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker```
**Describe alternatives you've considered**
It is easy enough to download the file into a temporary directory and read it from there. But if the dataset is large, it would be preferable to stream it progressively into memory, rather than spending time and space making an unnecessary copy to disk.
This is not necessarily something that should be within the scope of DataFusion, but since object stores are already supported I think it makes sense to support arbitrary URLs for consistency and completeness.
Contributor guide
Research direction
Start by tracing the DataFrame::read_* entry points and the existing object-store handling for supported sources. Determine how arbitrary HTTP URLs could be streamed, then verify that CSV and JSON URLs work without the current confusing error and that large inputs do not require a temporary disk copy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100