NVIDIA / NVIDIA/cudf

UInt128 handling in cudf-polars only covers one specific polars rewrite shape

Open
#24,108 1 comment 0 reactions 0 assignees View on GitHub
cudf-polars Needs Triage
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

polars 1.43+ rewrites `concat(a, b, ...).select(len())` into `col("len").cast(UInt128).sum().cast(IDX_DTYPE)`. libcudf has no 128-bit integer type (`size_type` is 32-bit, #13159), so `UInt128` fails to translate. The sum never actually needs 128 bits, it's just widened so accumulation can't overflow, then narrowed back to `IDX_DTYPE` before materializing. No GPU-resident table holds anywhere near `2**64` rows, so cudf-polars represents it as `UInt64` instead.

The fix (`translate.py`) only recognizes this one shape: an `Agg` named `"sum"`, or a `Cast` whose child is `Column("len")`. Any other `UInt128` usage still hits the generic `NotImplementedError` in `datatype.py`. If polars restructures this rewrite or introduces `UInt128` elsewhere, the new shape won't match, and the query silently loses GPU acceleration instead of running there.

Also: polars' own `Series.to_arrow()` can't export real `UInt128` data (`ArrowInvalid: Invalid or unsupported format string: '_plu128'`), so the dtype isn't fully Arrow-interoperable in polars itself yet either.

Contributor guide

Open the contributing guide

Research direction

Start in translate.py by reading the existing UInt128 handling for the sum and col("len"). Then inspect the generic NotImplementedError path in datatype.py and reproduce the polars 1.43+ concat(a, b, ...).select(len()) rewrite. Done means UInt128 rewrites beyond the one recognized shape remain GPU-executable where supported, while genuinely unsupported uses still fail explicitly.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.