NVIDIA / NVIDIA/cudf

[FEA] Support Polars bigidx by enforcing `IDX_DTYPE`

Open
#21,532 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
Currently, cudf-polars is inconsistent in how it handles the data types for intermediate sizes, lengths, and counts. In some places, we hardcode `UInt32`, but with the bigidx feature, the type would be `UInt64`.

When Polars is installed with the 64-bit runtime, its native index type becomes `UInt64`. Because our usage of index types is fragmented, these hardcoded 32-bit operations cause schema validation issues. Note: This is separate from the physical 32-bit size_type limit in libcudf tracked in #13159.

**Describe the solution you'd like**
Define a global `IDX_DTYPE` which should resolve at runtime. Eg.
```python
HAS_POLARS_RT_64 = pl.config.plr.RUNTIME_REPR == "rt64"
IDX_DTYPE = pl.UInt64 if HAS_POLARS_RT_64 else pl.UInt32
```

Then audit the codebase and prefer it where appropriate.

**Describe alternatives you've considered**
Leave the status quo alone until libcudf fully implements a 64-bit `size_type` (#13159). Fixing our schema translation does not actually unlock true 64-bit execution. Ie. we'll still fail if an operation exceeds the `size_type` limit.

#13159 is much larger issue and will affect the entire python stack (pylibcudf, cudf, cudf-polars). Additionally, we'd also need to consider how packaging would work? Example: `polars-u64-idx` would need a 64-bit build of libcudf. It's probably better to decouple 64-bit schema correctness from _true_ 64-bit execution for now.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.