Add helper functions from `DataFrame` to `LogicalPlanBuilder`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I'm trying to remove some explicit `DataFrame` references in `delta-rs`' codebase to operate directly on Logical Plans instead, so we can stream and materialize later in the process. Most methods on `DataFrame` are a wrapper around `LogicalPlanBuilder`, except for the `with_column`function.
### Describe the solution you'd like
I propose that we add a new trait, `LogicalPlanBuilderExt`, that adds convenience methods `with_column` (and `select`, etc) that performs the exact operations that `DataFrame` is using.
https://github.com/apache/datafusion/blob/d13d89129b77d8d251e3ac09e95009340ad3ab72/datafusion/core/src/dataframe/mod.rs#L2135-L2187
This also helps `DataFrame` become a loose wrapper around `LogicalPlanBuilder` and lets the logic live in `datafusion_expr`.
### Describe alternatives you've considered
N/A
### Additional context
N/A
Contributor guide
Research direction
Read the DataFrame helper implementations referenced in datafusion/core/src/dataframe/mod.rs, especially the linked method range, and compare them with LogicalPlanBuilder. Define the scope of LogicalPlanBuilderExt, including with_column and the other proposed helpers, then verify that the operations match DataFrame behavior and that the logic can live in datafusion_expr.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100