UInt128 handling in cudf-polars only covers one specific polars rewrite shape
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
polars 1.43+ rewrites `concat(a, b, ...).select(len())` into `col("len").cast(UInt128).sum().cast(IDX_DTYPE)`. libcudf has no 128-bit integer type (`size_type` is 32-bit, #13159), so `UInt128` fails to translate. The sum never actually needs 128 bits, it's just widened so accumulation can't overflow, then narrowed back to `IDX_DTYPE` before materializing. No GPU-resident table holds anywhere near `2**64` rows, so cudf-polars represents it as `UInt64` instead.
The fix (`translate.py`) only recognizes this one shape: an `Agg` named `"sum"`, or a `Cast` whose child is `Column("len")`. Any other `UInt128` usage still hits the generic `NotImplementedError` in `datatype.py`. If polars restructures this rewrite or introduces `UInt128` elsewhere, the new shape won't match, and the query silently loses GPU acceleration instead of running there.
Also: polars' own `Series.to_arrow()` can't export real `UInt128` data (`ArrowInvalid: Invalid or unsupported format string: '_plu128'`), so the dtype isn't fully Arrow-interoperable in polars itself yet either.
Contributor guide
Research direction
Start in translate.py by reading the existing UInt128 handling for the sum and col("len"). Then inspect the generic NotImplementedError path in datatype.py and reproduce the polars 1.43+ concat(a, b, ...).select(len()) rewrite. Done means UInt128 rewrites beyond the one recognized shape remain GPU-executable where supported, while genuinely unsupported uses still fail explicitly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100