apache / apache/datafusion

SQL on DataFrames

Aperta
#4,889 1 commento 1 reazione 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

At present, it is not very ergonomic to compose SQL. Say one wants to design a non-trivial pipeline with multiple stages by composing functions that perform various transformations on a DataFrame. This is only practical with the DataFrame API right now, by passing around partially transformed DataFrames and applying more transformations at each stage.

**Describe the solution you'd like**

Simply the ability to run SQL on an existing DataFrame (`DataFrame::sql`), so that a user always has the option to choose between SQL and the DataFrame API in more complex pipelines.

I'd suggest registering a temporary table reference with a name like `self`.

**Describe alternatives you've considered**

It is might be technically possible to do this by registering intermediate views. However,
* This would only work by staying within SQL through the whole pipeline, since there doesn't seem to be an API to create a view of a DataFrame either.
* It would require passing around a reference to `SessionContext` everywhere.
* Naming intermediate views, making sure they are globally unique, and passing around the names between functions as reference, which can be quite error-prone.
* Dropping (garbage collecting) views when they are no longer needed.

To be fair, other similar query engines do not have support for this either and have a similar behaviour. In Spark there is the `DataFrame.createGlobalTempView` method, which is a bit more helpful but still means dealing with globally unique naming.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

La issue nomina un punto di ingresso DataFrame::sql proposto e discute di SessionContext e delle viste temporanee, ma non include file né test. Inizia tracciando le API esistenti di DataFrame e SQL, quindi determina come sia possibile interrogare un DataFrame esistente senza che il chiamante debba gestire nomi di viste o un contesto; il lavoro sarà completo quando includerà una progettazione API chiara e una copertura per comporre SQL con trasformazioni DataFrame.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust, sql
Ambito
backend-api-design, databases
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.