NVIDIA / NVIDIA/cudf

[BUG] `.join(how="full")` on key column requiring upcast returns too many rows in cudf_polars with rapidsmpf

Open
#22,085 1 comment 0 reactions 0 assignees View on GitHub
bug cudf-polars
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

```python
In [1]: from functools import partialmethod
...: import polars
...: from cudf_polars.utils.config import Runtime, StreamingFallbackMode
...: executor = "streaming"
...: executor_options: dict[str, Any] = {}
...: executor_options["max_rows_per_partition"] = 5
...: executor_options["target_partition_size"] = 10
...: # We expect many tests to fall back, so silence the warnings
...: executor_options["fallback_mode"] = StreamingFallbackMode.SILENT
...: executor_options["runtime"] = Runtime.RAPIDSMPF
...: collect = polars.LazyFrame.collect
...: engine = polars.GPUEngine(executor=executor, executor_options=executor_options)
...: polars.LazyFrame.collect = partialmethod(collect, engine=engine)
...:
...: import polars as pl
...: swap = True
...: dtypes = ["Int64", "UInt8", "Int64"]
...: supertype, ltype, rtype = (getattr(pl, x) for x in dtypes)
...: ltype, rtype = (rtype, ltype) if swap else (ltype, rtype)
...:
...: left = pl.select(pl.Series("a", [1, 1, 3]).cast(ltype)).lazy()
...: right = pl.select(pl.Series("a", [1]).cast(rtype), b=pl.lit("A")).lazy()
...: left.join(right, on="a", how="full").collect()
Out[1]:
shape: (4, 3)
┌──────┬─────────┬──────┐
│ a ┆ a_right ┆ b │
│ --- ┆ --- ┆ --- │
│ i64 ┆ u8 ┆ str │
╞══════╪═════════╪══════╡
│ 1 ┆ null ┆ null │
│ 1 ┆ null ┆ null │
│ 3 ┆ null ┆ null │
│ null ┆ 1 ┆ A │
└──────┴─────────┴──────┘
```

while Polars returns

```python
In [1]: import polars as pl
...: swap = True
...: dtypes = ["Int64", "UInt8", "Int64"]
...: supertype, ltype, rtype = (getattr(pl, x) for x in dtypes)
...: ltype, rtype = (rtype, ltype) if swap else (ltype, rtype)
...:
...: left = pl.select(pl.Series("a", [1, 1, 3]).cast(ltype)).lazy()
...: right = pl.select(pl.Series("a", [1]).cast(rtype), b=pl.lit("A")).lazy()
...: left.join(right, on="a", how="full").collect()
Out[1]:
shape: (3, 3)
┌─────┬─────────┬──────┐
│ a ┆ a_right ┆ b │
│ --- ┆ --- ┆ --- │
│ i64 ┆ u8 ┆ str │
╞═════╪═════════╪══════╡
│ 1 ┆ 1 ┆ A │
│ 1 ┆ 1 ┆ A │
│ 3 ┆ null ┆ null │
└─────┴─────────┴──────┘
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.