apache / apache/datafusion

SQL on DataFrames

Open
#4,889 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

At present, it is not very ergonomic to compose SQL. Say one wants to design a non-trivial pipeline with multiple stages by composing functions that perform various transformations on a DataFrame. This is only practical with the DataFrame API right now, by passing around partially transformed DataFrames and applying more transformations at each stage.

**Describe the solution you'd like**

Simply the ability to run SQL on an existing DataFrame (`DataFrame::sql`), so that a user always has the option to choose between SQL and the DataFrame API in more complex pipelines.

I'd suggest registering a temporary table reference with a name like `self`.

**Describe alternatives you've considered**

It is might be technically possible to do this by registering intermediate views. However,
* This would only work by staying within SQL through the whole pipeline, since there doesn't seem to be an API to create a view of a DataFrame either.
* It would require passing around a reference to `SessionContext` everywhere.
* Naming intermediate views, making sure they are globally unique, and passing around the names between functions as reference, which can be quite error-prone.
* Dropping (garbage collecting) views when they are no longer needed.

To be fair, other similar query engines do not have support for this either and have a similar behaviour. In Spark there is the `DataFrame.createGlobalTempView` method, which is a bit more helpful but still means dealing with globally unique naming.

Contributor guide

Open the contributing guide

Research direction

The issue names a proposed DataFrame::sql entry point and discusses SessionContext and temporary views, but no files or tests. Start by tracing the existing DataFrame and SQL APIs, then determine how an existing DataFrame can be queried without caller-managed view names or context; done should include a clear API design and coverage for composing SQL with DataFrame transformations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.