NVIDIA / NVIDIA/cudf

[BUG] libcudf conda packages are shipping dependencies in the package

Open
#13,230 4 comments 0 reactions 1 assignee Claimed by @bdice View on GitHub
bug CMake libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Currently, libcudf conda packages are shipping libraries/headers from libcudf dependencies. In particular:

libcudf's conda package is 548 MB (all reported sizes are unzipped). It currently ships libraries/headers from:
- kvikio (< 1 MB)
- libcudacxx (5.2 MB)
- nvbench (4.1 MB)
- nvcomp (46 MB)

**Expected behavior**
libcudf conda packages should not ship its dependencies, but should instead list other conda packages as dependencies. This will prevent files from outside libcudf from being shipped in libcudf packages.

### Specific proposals for resolution

**kvikio (easy)**
kvikio should be easy to fix. RAPIDS already produces conda packages of kvikio and just need to use those, rather than letting it be found via CPM and repackaged in libcudf.

✔️ Resolved in: https://github.com/rapidsai/cudf/pull/13231

**libcudacxx (medium difficulty)**
`libcudacxx` is part of CCCL, along with Thrust and CUB. Historically, librmm has shipped most of the CCCL headers in `include/rapids` because they've been found by CPM during the librmm build and repackaged there. However, this is not ideal. libcudacxx is an exception to librmm shipping a RAPIDS-vendored CCCL because it was not needed by librmm -- only libcudf. I suspect we're repackaging libcudacxx in many RAPIDS packages right now. The fix here is to adopt [rapids-core-dependencies](https://github.com/rapidsai/rapids-cmake/tree/branch-23.06/conda/recipes/rapids_core_dependencies) across all of RAPIDS, which should be mostly ready for use, if I recall correctly.

**nvbench (very difficult)**
Ideally, `nvbench` should be its own conda package, but we often require patched versions. Therefore, we should consider moving nvbench headers to a special subdirectory `include/rapids` as we do for CCCL (see https://github.com/rapidsai/rapids-cmake/pull/98). We should also do a more granular review to decide if (and where) we want to be shipping files like `bin/nvbench-ctl` or `lib/objects-Release/nvbench.main/main.cu.o`.

✔️ Resolved in: https://github.com/rapidsai/cudf/pull/14399

**nvcomp (medium difficulty)**
`nvcomp` should be shipping in a separate conda package (probably on either the rapidsai or nvidia channels). Then we make it a dependency of libcudf. It is the largest component of libcudf that is being repackaged (around 10% of libcudf's size).

✔️ Resolved in: https://github.com/rapidsai/cudf/pull/13566

cc: @vuule (with whom I discussed this)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.