NVIDIA / NVIDIA/cudf

Follow-ups for rapidsmpf to cudf migration

Open
#22,771 0 comments 0 reactions 0 assignees View on GitHub
improvement libcudf rapids-mpf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

In addressing https://github.com/rapidsai/cudf/issues/22528 I focused on doing a wholesale migration of the cudf-specific rapidsmpf code into this repository. There are many changes that we might want to make that I did not want to lump into that PR to minimize disruptions and reduce the chance for surprising errors. This issue collects some of those ideas:
- [ ] https://github.com/rapidsai/cudf/issues/22769
- [x] ~Remove unnecessary environment from testing https://github.com/rapidsai/cudf/pull/22747#discussion_r3351060925~ no longer relevant since https://github.com/rapidsai/cudf/pull/22864 actually adds testing with ucxx and MPI
- [x] Add `cudf_streaming/` and `rapidsmpf/` to `.clang-format` `IncludeCategories` so that project headers get proper priority ordering (currently they fall to priority 9 "other system includes" and clang-format does not enforce the expected grouping). See https://github.com/rapidsai/cudf/pull/22747#discussion_r3351661351: https://github.com/rapidsai/cudf/pull/22881
- [x] Consider using smaller timeouts for all the test jobs https://github.com/rapidsai/cudf/pull/22747#discussion_r3344137304 and https://github.com/rapidsai/cudf/pull/22747#discussion_r3350958574: https://github.com/rapidsai/cudf/pull/22880
- [ ] Remove redundancy in namespace names. The namespaces under `cudf_streaming` like `streaming` and `integration` are copied from `rapidsmpf`, and they don't fit well here. https://github.com/rapidsai/cudf/pull/22747#discussion_r3351037105
- [x] Remove warning suppressions if possible https://github.com/rapidsai/rapidsmpf/pull/1083#discussion_r3370651625: https://github.com/rapidsai/rapidsmpf/pull/1100
- [x] Remove `spill/unspill_partitions` from `partition.hpp` and move them back to rapidsmpf https://github.com/rapidsai/cudf/pull/22810
- [x] Reenable benchmarks in devcontainer builds (see https://github.com/rapidsai/cudf/pull/22814#discussion_r3383796528). To do that, we will need to introduce separate flags for turning MPI and UCXX on/off. We cannot enable MPI in pip devcontainers since rapidsmpf wheels do not support them, but we can enable them in conda devcontainers, so once we have separate CMake flags we can turn them on/off independently: https://github.com/rapidsai/cudf/pull/22853 and https://github.com/rapidsai/cudf/pull/22860
- [ ] Remove all unnecessary testing helpers https://github.com/rapidsai/cudf/pull/22814#pullrequestreview-4462369025
- [ ] Add cudf streaming distributed python tests. Currently it only has single rank tests

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.