[BUG] `pl.LazyFrame[length=5].unique().tail(4)` returns 5 rows with rapidsmpf w/ <=4 max_rows_per_partition
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
This appears to more generally fail when `executor_options["max_rows_per_partition"] <= tail`
```python
In [1]: from functools import partialmethod
...: import polars
...: from cudf_polars.utils.config import Runtime, StreamingFallbackMode
...: executor = "streaming"
...: executor_options: dict[str, Any] = {}
...: executor_options["max_rows_per_partition"] = 4
...: executor_options["target_partition_size"] = 10
...: # We expect many tests to fall back, so silence the warnings
...: executor_options["fallback_mode"] = StreamingFallbackMode.SILENT
...: executor_options["runtime"] = Runtime.RAPIDSMPF
...: collect = polars.LazyFrame.collect
...: engine = polars.GPUEngine(executor=executor, executor_options=executor_options)
...: polars.LazyFrame.collect = partialmethod(collect, engine=engine)
...:
...: import polars as pl
...: tail = 4
...: df = pl.DataFrame(
...: {
...: "x": [1, 2, 3, 4, 5],
...: }
...: )
...: df.lazy().unique().tail(tail).collect()
Out[1]:
shape: (5, 1)
┌─────┐
│ x │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
│ 4 │
│ 5 │
│ 3 │
└─────┘
```
vs
```python
In [1]: import polars as pl
...: tail = 4
...: df = pl.DataFrame(
...: {
...: "x": [1, 2, 3, 4, 5],
...: }
...: )
...: df.lazy().unique().tail(tail).collect()
Out[1]:
shape: (4, 1)
┌─────┐
│ x │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 5 │
│ 3 │
│ 4 │
└─────┘
```
Contributor guide
Assessment
This issue has not been assessed yet.