[FEA] Create `cudf::utilities::` namespace to expose internal utilities to downstream libraries
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## **Is your feature request related to a problem? Please describe.**
Currently, `libcudf` encapsulates its internal implementation details within the `cudf::detail::` namespace. By design, APIs within `detail` are not part of the public API and do not offer stability guarantees. However, because there is no officially supported public utility namespace, downstream applications across the RAPIDS/NVIDIA ecosystem — such as `spark-rapids-jni`, frequently need access to these internal tools to avoid duplicating complex C++/CUDA logic. For example, it frequently relies on internals like `cudf/detail/utilities/pinned_host_vector.hpp` and functions such as `cudf::detail::copy_if_safe`.
## **Describe the solution you'd like**
**We propose creating a new, formally exposed sub-namespace: `cudf::utilities::` (or similar).**
This namespace would host stable, reusable utility code that is valuable to both `libcudf` internally and downstream RAPIDS libraries. By migrating commonly shared utilities out of `cudf::detail::` (or `cudf::detail::utilities::`) and into `cudf::utilities::`, we can:
1. Provide a stable, public-facing API for downstream ecosystem consumers.
2. Prevent code duplication across RAPIDS repositories.
3. Preserve the strict internal-only contract of the `cudf::detail::` namespace.
**In addition to adding a new namespace, we should also re-organize the cudf headers.**
Currently, the headers are organized as:
```
cpp/include
├── cudf/
│ ├── ...
│ ├── detail
│ ├── ...
├── cudf_test/
└── nvtext/
```
That means the `details` folder containing `cudf::detail::` headers is stored inside the public `include/cudf` folder. This would lead to the issue such that when installing cudf library these headers are also installed together, providing the users the opportunity to using them by the wrong way that violates our encapsulation principle.
We could do better by moving the folder containing the details headers into a separate folder under `include` and rename it to `cudf_detail`, such as:
```
cpp/include
├── cudf/
│ ├── ...
│ ├── utilities
│ ├── ...
├── cudf_detail/
├── cudf_test/
└── nvtext/
```
By doing so, installing the `include/cudf` headers will not contain any details header.
**Beyond moving the headers, we should also consider renaming the `cudf::detail::` namespace into `cudf::internal` namespace to strengthen its intention: they are only used internally for cudf development.**
## **Describe alternatives you've considered**
If this proposal is not acceptable, we have no choice but:
* **Status Quo**: Allow downstream libraries to continue relying on `cudf::detail::`. This contradict with what we are trying to do (preventing downstream apps/libraries from using `cudf::detail::`).
* **Code Duplication**: Force downstream apps/libraries to implement and maintain their own versions of these utilities, increasing the maintenance burden and leading to codebase fragmentation. *Functions like `cudf::detail::copy_if` have to be duplicate by this way.*
* **Promote detail APIs directly to `cudf::`**: Exposing these utilities directly in the top-level `cudf::` namespace might clutter the primary dataframe API with lower-level system and memory utilities. *Not all detail APIs can be promoted due to various constraints.*
## **Additional context**
This request aims to improve cross-ecosystem maintainability. Once the `cudf::utilities::` namespace is established, we can progressively open PRs to migrate the most highly reusable components (like `cudf::detail::copy_if`) and subsequently update downstream libraries to use the safe public paths.
Contributor guide
Assessment
This issue has not been assessed yet.