apache / apache/datafusion

Add helper functions from `DataFrame` to `LogicalPlanBuilder`

Open
#19,564 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I'm trying to remove some explicit `DataFrame` references in `delta-rs`' codebase to operate directly on Logical Plans instead, so we can stream and materialize later in the process. Most methods on `DataFrame` are a wrapper around `LogicalPlanBuilder`, except for the `with_column`function.

### Describe the solution you'd like

I propose that we add a new trait, `LogicalPlanBuilderExt`, that adds convenience methods `with_column` (and `select`, etc) that performs the exact operations that `DataFrame` is using.

https://github.com/apache/datafusion/blob/d13d89129b77d8d251e3ac09e95009340ad3ab72/datafusion/core/src/dataframe/mod.rs#L2135-L2187

This also helps `DataFrame` become a loose wrapper around `LogicalPlanBuilder` and lets the logic live in `datafusion_expr`.

### Describe alternatives you've considered

N/A

### Additional context

N/A

Contributor guide

Open the contributing guide

Research direction

Read the DataFrame helper implementations referenced in datafusion/core/src/dataframe/mod.rs, especially the linked method range, and compare them with LogicalPlanBuilder. Define the scope of LogicalPlanBuilderExt, including with_column and the other proposed helpers, then verify that the operations match DataFrame behavior and that the logic can live in datafusion_expr.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.