[FEA] Support streaming ``shift``/``diff`` expressions in cuDF-Polars
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Row-order-sensitive expressions such as plain `shift()` are supported by the GPU engine in single-partition/in-memory execution, but are not supported by streaming multi-partition execution.
This issue should cover the common row-order expression variants that need boundary state across streaming partitions:
- [ ] `shift`
- [ ] `diff`
- [ ] `pct_change`
- [ ] `forward_fill`
- [ ] `backward_fill`
**Example**:
```python
def test_expr_shift_streaming(engine: GPUEngine) -> None:
"""``shift`` on the full frame.
Status
------
- In-memory OK
- Streaming NOT supported
"""
lf = pl.LazyFrame({"x": [1, 2, 3, 4, 5]})
q = lf.select(
pl.col("x").shift(1).alias("lag_1"),
pl.col("x").shift(-1).alias("lead_1"),
)
assert_gpu_result_equal(q, engine=engine)
```
Contributor guide
Research direction
Start with the test_expr_shift_streaming example and the assert_gpu_result_equal helper, then trace how GPUEngine handles streaming multi-partition expressions. Done means streaming execution matches in-memory results for shift, diff, pct_change, forward_fill, and backward_fill, including the shown positive and negative shifts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100