[Story][FEA] Streaming ordered-window execution in cuDF-Polars
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
This issue tracks cuDF-Polars support for ordered/window operations that require boundary state across streaming partitions/ranks.
This issue is intentionally scoped to non-grouped streaming execution. Grouped range windows, grouped `over(...)` expressions, ordered grouped reductions, and other related feature families are tracked separately below.
## Active Scope
- Streaming `LazyFrame.rolling(index_column=...).agg(...)`
- Streaming expression rolling, if this needs separate tracking from `LazyFrame.rolling(...)`
- Plain streaming row-order expressions such as `shift()`, `diff()`, `pct_change()`, and forward/backward fill
- Boundary/halo exchange between neighboring partitions/ranks
- Preserving ordered partition semantics across streaming execution
## Active Tracking Issues
| Feature | Status | Tracking |
|---|---|---|
| `LazyFrame.rolling(index_column=...).agg(...)` | In-memory supported; streaming multi-partition missing | #22033 |
| Single-rank streaming `LazyFrame.rolling(...)` | Open PR | #22308 |
| Plain streaming row-order expressions | Streaming support missing | #23593 |
## Representative Reproducers
### Streaming `LazyFrame.rolling(...)`
Pytest
```python
@pytest.mark.parametrize("closed", ["right", "left", "both", "none"])
def test_lazyframe_rolling_non_grouped(engine: GPUEngine, closed: str) -> None:
lf = pl.LazyFrame(
{
"idx": [1, 2, 3, 4, 5, 6, 7, 8],
"val": [10, 20, 30, 40, 50, 60, 70, 80],
}
)
q = lf.rolling("idx", period="3i", closed=closed).agg(
s=pl.col("val").sum(),
m=pl.col("val").min(),
x=pl.col("val").max(),
n=pl.len(),
)
assert_gpu_result_equal(q, engine=engine)
```
### Plain streaming row-order expressions
Pytest
```python
def test_streaming_shift_diff(engine: GPUEngine) -> None:
lf = pl.LazyFrame({"x": [1, 2, 3, 4, 5]})
q = lf.select(
pl.col("x").shift(1).alias("lag_1"),
pl.col("x").shift(-1).alias("lead_1"),
pl.col("x").diff().alias("diff_1"),
)
assert_gpu_result_equal(q, engine=engine)
```
## Related Work Tracked Elsewhere
| Feature family | Tracking |
|---|---|
| Broad rolling-operation umbrella | #18633 |
| Grouped range-window support | #23594 |
| Grouped `LazyFrame.rolling(..., group_by=...)` | #23590 |
| `rolling_*_by(...).over(...)` | #23591 |
| `group_by_dynamic(...)` | #17144 |
| Ordered first/last grouped reductions | #23592 |
| Grouped/nested `shift`/`diff` aggregation forms | #19934 |
| `join_asof` | #24110 |
| Grouped `pl.corr` | #22908 |
| Grouped list aggregation and explode | #23595 |
| Broad unsupported-expression inventory | #23151 |
## Completed Or Moved Out Of Scope
These items were previously listed here, but are now either supported or tracked by more focused issues:
- Simple grouped `over(...)` aggregations are supported ✅
- Ordered `cum_sum().over(...)` is supported ✅
- Direct `shift(...).over(...)` is supported by #23451 and #23466 ✅
- Direct `diff(...).over(...)` is supported by #23497 ✅
- Fixed-size rolling expressions inside `over(...)`, such as `rolling_mean(window_size=2).over("g")`, are supported by #23468 ✅
- Grouped range-window features moved to #23594.
- Ordered grouped reductions moved to #23592.
Contributor guide
Assessment
This issue has not been assessed yet.