SQL on DataFrames
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
At present, it is not very ergonomic to compose SQL. Say one wants to design a non-trivial pipeline with multiple stages by composing functions that perform various transformations on a DataFrame. This is only practical with the DataFrame API right now, by passing around partially transformed DataFrames and applying more transformations at each stage.
**Describe the solution you'd like**
Simply the ability to run SQL on an existing DataFrame (`DataFrame::sql`), so that a user always has the option to choose between SQL and the DataFrame API in more complex pipelines.
I'd suggest registering a temporary table reference with a name like `self`.
**Describe alternatives you've considered**
It is might be technically possible to do this by registering intermediate views. However,
* This would only work by staying within SQL through the whole pipeline, since there doesn't seem to be an API to create a view of a DataFrame either.
* It would require passing around a reference to `SessionContext` everywhere.
* Naming intermediate views, making sure they are globally unique, and passing around the names between functions as reference, which can be quite error-prone.
* Dropping (garbage collecting) views when they are no longer needed.
To be fair, other similar query engines do not have support for this either and have a similar behaviour. In Spark there is the `DataFrame.createGlobalTempView` method, which is a bit more helpful but still means dealing with globally unique naming.
Contributor guide
Research direction
The issue names a proposed DataFrame::sql entry point and discusses SessionContext and temporary views, but no files or tests. Start by tracing the existing DataFrame and SQL APIs, then determine how an existing DataFrame can be queried without caller-managed view names or context; done should include a clear API design and coverage for composing SQL with DataFrame transformations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100