Read DataFrame from URL
- Lingua principale
- Rust
- Stelle
- 9.3k
- Fork
- 2.4k
- Merge medio
- 3g 11h
- PR unite (30g)
- 360
Descrizione
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
It is a common use-case to dynamically download datasets (usually CSV or JSON) from third-party APIs, particularly when the data changes with time.
**Describe the solution you'd like**
It would be nice to be able to pass any URL to the `DataFrame::read_*` and stream the data in. Currently a confusing error is displayed if the URL doesn't correspond to a supported source like S3.
```Error: Internal error: No suitable object store found for https://<...>.csv. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker```
**Describe alternatives you've considered**
It is easy enough to download the file into a temporary directory and read it from there. But if the dataset is large, it would be preferable to stream it progressively into memory, rather than spending time and space making an unnecessary copy to disk.
This is not necessarily something that should be within the scope of DataFusion, but since object stores are already supported I think it makes sense to support arbitrary URLs for consistency and completeness.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia tracciando gli entry point DataFrame::read_* e la gestione esistente degli object store per le sorgenti supportate. Determina come poter effettuare lo streaming di URL HTTP arbitrari, quindi verifica che gli URL CSV e JSON funzionino senza l’errore attualmente fuorviante e che gli input di grandi dimensioni non richiedano una copia temporanea su disco.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- rust
- Ambito
- data
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 25/100