NVIDIA / NVIDIA/cudf

[BUG] `pl.LazyFrame[length=5].unique().tail(4)` returns 5 rows with rapidsmpf w/ <=4 max_rows_per_partition

Open
#22,075 1 comment 0 reactions 0 assignees View on GitHub
bug cudf-polars
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

This appears to more generally fail when `executor_options["max_rows_per_partition"] <= tail`

```python
In [1]: from functools import partialmethod
...: import polars
...: from cudf_polars.utils.config import Runtime, StreamingFallbackMode
...: executor = "streaming"
...: executor_options: dict[str, Any] = {}
...: executor_options["max_rows_per_partition"] = 4
...: executor_options["target_partition_size"] = 10
...: # We expect many tests to fall back, so silence the warnings
...: executor_options["fallback_mode"] = StreamingFallbackMode.SILENT
...: executor_options["runtime"] = Runtime.RAPIDSMPF
...: collect = polars.LazyFrame.collect
...: engine = polars.GPUEngine(executor=executor, executor_options=executor_options)
...: polars.LazyFrame.collect = partialmethod(collect, engine=engine)
...:
...: import polars as pl
...: tail = 4
...: df = pl.DataFrame(
...: {
...: "x": [1, 2, 3, 4, 5],
...: }
...: )
...: df.lazy().unique().tail(tail).collect()
Out[1]:
shape: (5, 1)
┌─────┐
│ x │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
│ 4 │
│ 5 │
│ 3 │
└─────┘
```

vs

```python
In [1]: import polars as pl
...: tail = 4
...: df = pl.DataFrame(
...: {
...: "x": [1, 2, 3, 4, 5],
...: }
...: )
...: df.lazy().unique().tail(tail).collect()
Out[1]:
shape: (4, 1)
┌─────┐
│ x │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 5 │
│ 3 │
│ 4 │
└─────┘
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.