NVIDIA / NVIDIA/cudf

[FEA] Use more performant dtype introspection utility in `select_dtypes`

Open
#18,219 0 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

xref https://github.com/rapidsai/cudf/pull/18141#discussion_r1978663659

`select_dtypes` has a internal dtype introspection function `cudf_dtype_from_pydata_dtype` to "normalize" `include=` and `exclude=` dtypes https://github.com/rapidsai/cudf/blob/8e4408c19a5a8cd84989c39f16f0a07e510c4824/python/cudf/cudf/core/dataframe.py#L6857

This function is a bit more expensive than other dtype checking functions in cudf and has a drastic performance difference when normalizing string-specified types vs dtype objects. At minimum some caching should be implemented to not incur such a performance penalty between the two

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.