apache / apache/datafusion

Dataframe partition method

Open
#23,252 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently I use do a couple of `dataframe.filter` calls, when a single pass might be more efficient
```
let matches_df = df.clone().filter(expr.clone());
let inverse_matches_df = df.clone().filter(expr.is_not_true());
```

It could be nice if there was a function `let (matches_df, inverse_matches_df) = df.partition(expr);`
if it could produce both dataframes in a single pass from the filter expression.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

The running two filters does actually work, but a dedicated function might be clearer and could be more efficient.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the dataframe.filter entry point and the expression handling it uses. Compare the two filter calls in the issue and determine how partition should return matches and inverse matches in one pass. Done means the new API produces both dataframes with the requested expression semantics and has coverage for the relevant cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.