[FEA] Use more performant dtype introspection utility in `select_dtypes`
Open
feature request
Python
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
xref https://github.com/rapidsai/cudf/pull/18141#discussion_r1978663659
`select_dtypes` has a internal dtype introspection function `cudf_dtype_from_pydata_dtype` to "normalize" `include=` and `exclude=` dtypes https://github.com/rapidsai/cudf/blob/8e4408c19a5a8cd84989c39f16f0a07e510c4824/python/cudf/cudf/core/dataframe.py#L6857
This function is a bit more expensive than other dtype checking functions in cudf and has a drastic performance difference when normalizing string-specified types vs dtype objects. At minimum some caching should be implemented to not incur such a performance penalty between the two
Contributor guide
Assessment
This issue has not been assessed yet.