SQL on DataFrames
- Lingua principale
- Rust
- Stelle
- 9.3k
- Fork
- 2.4k
- Merge medio
- 3g 11h
- PR unite (30g)
- 360
Descrizione
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
At present, it is not very ergonomic to compose SQL. Say one wants to design a non-trivial pipeline with multiple stages by composing functions that perform various transformations on a DataFrame. This is only practical with the DataFrame API right now, by passing around partially transformed DataFrames and applying more transformations at each stage.
**Describe the solution you'd like**
Simply the ability to run SQL on an existing DataFrame (`DataFrame::sql`), so that a user always has the option to choose between SQL and the DataFrame API in more complex pipelines.
I'd suggest registering a temporary table reference with a name like `self`.
**Describe alternatives you've considered**
It is might be technically possible to do this by registering intermediate views. However,
* This would only work by staying within SQL through the whole pipeline, since there doesn't seem to be an API to create a view of a DataFrame either.
* It would require passing around a reference to `SessionContext` everywhere.
* Naming intermediate views, making sure they are globally unique, and passing around the names between functions as reference, which can be quite error-prone.
* Dropping (garbage collecting) views when they are no longer needed.
To be fair, other similar query engines do not have support for this either and have a similar behaviour. In Spark there is the `DataFrame.createGlobalTempView` method, which is a bit more helpful but still means dealing with globally unique naming.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
La issue nomina un punto di ingresso DataFrame::sql proposto e discute di SessionContext e delle viste temporanee, ma non include file né test. Inizia tracciando le API esistenti di DataFrame e SQL, quindi determina come sia possibile interrogare un DataFrame esistente senza che il chiamante debba gestire nomi di viste o un contesto; il lavoro sarà completo quando includerà una progettazione API chiara e una copertura per comporre SQL con trasformazioni DataFrame.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- rust, sql
- Ambito
- backend-api-design, databases
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100