[FEA] Support Polars bigidx by enforcing `IDX_DTYPE`
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Currently, cudf-polars is inconsistent in how it handles the data types for intermediate sizes, lengths, and counts. In some places, we hardcode `UInt32`, but with the bigidx feature, the type would be `UInt64`.
When Polars is installed with the 64-bit runtime, its native index type becomes `UInt64`. Because our usage of index types is fragmented, these hardcoded 32-bit operations cause schema validation issues. Note: This is separate from the physical 32-bit size_type limit in libcudf tracked in #13159.
**Describe the solution you'd like**
Define a global `IDX_DTYPE` which should resolve at runtime. Eg.
```python
HAS_POLARS_RT_64 = pl.config.plr.RUNTIME_REPR == "rt64"
IDX_DTYPE = pl.UInt64 if HAS_POLARS_RT_64 else pl.UInt32
```
Then audit the codebase and prefer it where appropriate.
**Describe alternatives you've considered**
Leave the status quo alone until libcudf fully implements a 64-bit `size_type` (#13159). Fixing our schema translation does not actually unlock true 64-bit execution. Ie. we'll still fail if an operation exceeds the `size_type` limit.
#13159 is much larger issue and will affect the entire python stack (pylibcudf, cudf, cudf-polars). Additionally, we'd also need to consider how packaging would work? Example: `polars-u64-idx` would need a 64-bit build of libcudf. It's probably better to decouple 64-bit schema correctness from _true_ 64-bit execution for now.
Contributor guide
Assessment
This issue has not been assessed yet.