[BUG] libcudf conda packages are shipping dependencies in the package
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Currently, libcudf conda packages are shipping libraries/headers from libcudf dependencies. In particular:
libcudf's conda package is 548 MB (all reported sizes are unzipped). It currently ships libraries/headers from:
- kvikio (< 1 MB)
- libcudacxx (5.2 MB)
- nvbench (4.1 MB)
- nvcomp (46 MB)
**Expected behavior**
libcudf conda packages should not ship its dependencies, but should instead list other conda packages as dependencies. This will prevent files from outside libcudf from being shipped in libcudf packages.
### Specific proposals for resolution
**kvikio (easy)**
kvikio should be easy to fix. RAPIDS already produces conda packages of kvikio and just need to use those, rather than letting it be found via CPM and repackaged in libcudf.
✔️ Resolved in: https://github.com/rapidsai/cudf/pull/13231
**libcudacxx (medium difficulty)**
`libcudacxx` is part of CCCL, along with Thrust and CUB. Historically, librmm has shipped most of the CCCL headers in `include/rapids` because they've been found by CPM during the librmm build and repackaged there. However, this is not ideal. libcudacxx is an exception to librmm shipping a RAPIDS-vendored CCCL because it was not needed by librmm -- only libcudf. I suspect we're repackaging libcudacxx in many RAPIDS packages right now. The fix here is to adopt [rapids-core-dependencies](https://github.com/rapidsai/rapids-cmake/tree/branch-23.06/conda/recipes/rapids_core_dependencies) across all of RAPIDS, which should be mostly ready for use, if I recall correctly.
**nvbench (very difficult)**
Ideally, `nvbench` should be its own conda package, but we often require patched versions. Therefore, we should consider moving nvbench headers to a special subdirectory `include/rapids` as we do for CCCL (see https://github.com/rapidsai/rapids-cmake/pull/98). We should also do a more granular review to decide if (and where) we want to be shipping files like `bin/nvbench-ctl` or `lib/objects-Release/nvbench.main/main.cu.o`.
✔️ Resolved in: https://github.com/rapidsai/cudf/pull/14399
**nvcomp (medium difficulty)**
`nvcomp` should be shipping in a separate conda package (probably on either the rapidsai or nvidia channels). Then we make it a dependency of libcudf. It is the largest component of libcudf that is being repackaged (around 10% of libcudf's size).
✔️ Resolved in: https://github.com/rapidsai/cudf/pull/13566
cc: @vuule (with whom I discussed this)
Contributor guide
Assessment
This issue has not been assessed yet.