NVIDIA / NVIDIA/cudf

[FEA] Support streaming ``shift``/``diff`` expressions in cuDF-Polars

Open
#23,593 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Row-order-sensitive expressions such as plain `shift()` are supported by the GPU engine in single-partition/in-memory execution, but are not supported by streaming multi-partition execution.

This issue should cover the common row-order expression variants that need boundary state across streaming partitions:

- [ ] `shift`
- [ ] `diff`
- [ ] `pct_change`
- [ ] `forward_fill`
- [ ] `backward_fill`

**Example**:

```python
def test_expr_shift_streaming(engine: GPUEngine) -> None:
"""``shift`` on the full frame.

Status
------
- In-memory OK
- Streaming NOT supported
"""
lf = pl.LazyFrame({"x": [1, 2, 3, 4, 5]})
q = lf.select(
pl.col("x").shift(1).alias("lag_1"),
pl.col("x").shift(-1).alias("lead_1"),
)
assert_gpu_result_equal(q, engine=engine)
```

Contributor guide

Open the contributing guide

Research direction

Start with the test_expr_shift_streaming example and the assert_gpu_result_equal helper, then trace how GPUEngine handles streaming multi-partition expressions. Done means streaming execution matches in-memory results for shift, diff, pct_change, forward_fill, and backward_fill, including the shown positive and negative shifts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.