NVIDIA / NVIDIA/cudf

[Story][FEA] Streaming ordered-window execution in cuDF-Polars

Open
#22,032 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars feature request rapids-mpf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

This issue tracks cuDF-Polars support for ordered/window operations that require boundary state across streaming partitions/ranks.

This issue is intentionally scoped to non-grouped streaming execution. Grouped range windows, grouped `over(...)` expressions, ordered grouped reductions, and other related feature families are tracked separately below.

## Active Scope

- Streaming `LazyFrame.rolling(index_column=...).agg(...)`
- Streaming expression rolling, if this needs separate tracking from `LazyFrame.rolling(...)`
- Plain streaming row-order expressions such as `shift()`, `diff()`, `pct_change()`, and forward/backward fill
- Boundary/halo exchange between neighboring partitions/ranks
- Preserving ordered partition semantics across streaming execution

## Active Tracking Issues

| Feature | Status | Tracking |
|---|---|---|
| `LazyFrame.rolling(index_column=...).agg(...)` | In-memory supported; streaming multi-partition missing | #22033 |
| Single-rank streaming `LazyFrame.rolling(...)` | Open PR | #22308 |
| Plain streaming row-order expressions | Streaming support missing | #23593 |

## Representative Reproducers

### Streaming `LazyFrame.rolling(...)`

Pytest

```python
@pytest.mark.parametrize("closed", ["right", "left", "both", "none"])
def test_lazyframe_rolling_non_grouped(engine: GPUEngine, closed: str) -> None:
lf = pl.LazyFrame(
{
"idx": [1, 2, 3, 4, 5, 6, 7, 8],
"val": [10, 20, 30, 40, 50, 60, 70, 80],
}
)

q = lf.rolling("idx", period="3i", closed=closed).agg(
s=pl.col("val").sum(),
m=pl.col("val").min(),
x=pl.col("val").max(),
n=pl.len(),
)

assert_gpu_result_equal(q, engine=engine)
```

### Plain streaming row-order expressions

Pytest

```python
def test_streaming_shift_diff(engine: GPUEngine) -> None:
lf = pl.LazyFrame({"x": [1, 2, 3, 4, 5]})

q = lf.select(
pl.col("x").shift(1).alias("lag_1"),
pl.col("x").shift(-1).alias("lead_1"),
pl.col("x").diff().alias("diff_1"),
)

assert_gpu_result_equal(q, engine=engine)
```

## Related Work Tracked Elsewhere

| Feature family | Tracking |
|---|---|
| Broad rolling-operation umbrella | #18633 |
| Grouped range-window support | #23594 |
| Grouped `LazyFrame.rolling(..., group_by=...)` | #23590 |
| `rolling_*_by(...).over(...)` | #23591 |
| `group_by_dynamic(...)` | #17144 |
| Ordered first/last grouped reductions | #23592 |
| Grouped/nested `shift`/`diff` aggregation forms | #19934 |
| `join_asof` | #24110 |
| Grouped `pl.corr` | #22908 |
| Grouped list aggregation and explode | #23595 |
| Broad unsupported-expression inventory | #23151 |

## Completed Or Moved Out Of Scope

These items were previously listed here, but are now either supported or tracked by more focused issues:

- Simple grouped `over(...)` aggregations are supported ✅
- Ordered `cum_sum().over(...)` is supported ✅
- Direct `shift(...).over(...)` is supported by #23451 and #23466 ✅
- Direct `diff(...).over(...)` is supported by #23497 ✅
- Fixed-size rolling expressions inside `over(...)`, such as `rolling_mean(window_size=2).over("g")`, are supported by #23468 ✅
- Grouped range-window features moved to #23594.
- Ordered grouped reductions moved to #23592.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.