NVIDIA / NVIDIA/cudf

[BUG] computeColumn in 26.08.1 fails without NVRTC

Open
#23,903 1 comment 0 reactions 0 assignees View on GitHub
? - Needs Triage bug
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

Maven Central `ai.rapids:cudf:26.08.1` (`cuda13` classifier) throws an RTCX error on `CompiledExpression.computeColumn` when `libnvrtc.so.13` is not on the host. The same program succeeds with `ai.rapids:cudf:26.06.0` (`cuda13`).

`LIBCUDF_JIT_ENABLED` is unset (default `false`). This is the AST interpreter path, not an explicit JIT compile. Setting `LIBCUDF_JIT_ENABLED=false` does not avoid the failure.

```
ai.rapids.cudf.CudfException: RTCX failure at: /__w/cudf/cudf/.java-work/libcudf-build/_deps/rtcx-src/rtcx.cpp:285: Failed to load dynamic library `libnvrtc.so` (tried: libnvrtc.so.13)
at ai.rapids.cudf.ast.CompiledExpression.computeColumn(Native Method)
at ai.rapids.cudf.ast.CompiledExpression.computeColumn(CompiledExpression.java:77)
```

The classifier JAR does not ship NVRTC.

**Steps/Code to reproduce bug**

`Repro.java`:

```java
import ai.rapids.cudf.ColumnVector;
import ai.rapids.cudf.Table;
import ai.rapids.cudf.ast.ColumnReference;
import ai.rapids.cudf.ast.CompiledExpression;

public class Repro {
public static void main(String[] args) {
try (ColumnVector c = ColumnVector.fromInts(1, 2, 3);
Table t = new Table(c);
CompiledExpression e = new ColumnReference(0).compile();
ColumnVector r = e.computeColumn(t)) {
System.out.println("OK rows=" + r.getRowCount());
}
}
}
```

```bash
curl -fsSLO https://repo1.maven.org/maven2/ai/rapids/cudf/26.06.0/cudf-26.06.0-cuda13.jar
curl -fsSLO https://repo1.maven.org/maven2/ai/rapids/cudf/26.08.1/cudf-26.08.1-cuda13.jar
curl -fsSLO https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar

# 26.06.0 -> OK
docker run --rm --gpus all -v "$PWD":/workspace -w /workspace eclipse-temurin:17-jdk \
java -cp 'cudf-26.06.0-cuda13.jar:slf4j-api-1.7.30.jar' Repro.java

# 26.08.1 -> RTCX failure
docker run --rm --gpus all -v "$PWD":/workspace -w /workspace eclipse-temurin:17-jdk \
java -cp 'cudf-26.08.1-cuda13.jar:slf4j-api-1.7.30.jar' Repro.java
```

26.06.0:

```
OK rows=3
```

26.08.1:

```
Exception in thread "main" ai.rapids.cudf.CudfException: RTCX failure at: /__w/cudf/cudf/.java-work/libcudf-build/_deps/rtcx-src/rtcx.cpp:285: Failed to load dynamic library `libnvrtc.so` (tried: libnvrtc.so.13)
at ai.rapids.cudf.ast.CompiledExpression.computeColumn(Native Method)
at ai.rapids.cudf.ast.CompiledExpression.computeColumn(CompiledExpression.java:77)
at Repro.main(Repro.java:11)
```

**Expected behavior**

With `LIBCUDF_JIT_ENABLED` unset/`false`, `computeColumn` should use the AST interpreter and not require `libnvrtc`, matching 26.06.0.

**Environment overview (please complete the following information)**
- Environment location: Docker
- Method of cuDF install: Maven Central classifier JARs (`ai.rapids:cudf:26.06.0:cuda13` and `ai.rapids:cudf:26.08.1:cuda13`) plus declared `slf4j-api`

```bash
docker pull eclipse-temurin:17-jdk
docker run --rm --gpus all -v "$PWD":/workspace -w /workspace eclipse-temurin:17-jdk \
java -cp 'cudf-26.08.1-cuda13.jar:slf4j-api-1.7.30.jar' Repro.java
```

Host: x86_64 Linux, NVIDIA Container Toolkit (`--gpus all`). The JDK image has no CUDA toolkit and no `libnvrtc`.

**Environment details**
Not Applicable.

**Additional context**

In 26.08, libcudf replaced jitify with librtcx (https://github.com/rapidsai/cudf/pull/22654), and this update appears to have moved `libnvrtc` loading to the JIT cache initialization step (this previously happened during first kernel compilation).

In 26.06, cache initialization only built the jitify program cache

```cpp
void context::ensure_jit_cache_initialized()
{
std::call_once(_program_cache_init_flag,
[&]() { _program_cache = jit::program_cache::create(); });
}
```
https://github.com/NVIDIA/cudf/blob/v26.06.00/cpp/src/runtime/context.cpp#L27-L31

and did not load `libnvrtc`. My understanding is that jitify loads `libnvrtc` lazily when a kernel is actually compiled, so with `LIBCUDF_JIT_ENABLED` set to false, `libnvrtc` was never loaded.

In 26.08, the same function unconditionally initializes rtcx, and rtcx loads `libnvrtc` inside `initialize()`

```cpp
void context::ensure_jit_cache_initialized()
{
std::call_once(_jit_cache_init_flag, [&]() {
// make sure the required directories exist
std::filesystem::create_directories(_config.rtcx_cache_dir);
std::filesystem::create_directories(_config.jit_bundle_dir);
std::filesystem::create_directories(_config.jit_pch_dir);
std::filesystem::create_directories(_config.jit_tmp_dir);

rtcx::initialize();
...
```

https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/src/runtime/context.cpp#L65-L74

See `LibNVRTC::_load()` call in `rtcx::initialize()`:
https://github.com/rapidsai/librtcx/blob/efad266c1fd9de6d8486c6ba71bfa74df063eb1f/rtcx.cpp#L463-L474

`LibNVRTC::_load()` `dlopen`s `libnvrtc.so.`:
https://github.com/rapidsai/librtcx/blob/efad266c1fd9de6d8486c6ba71bfa74df063eb1f/rtcx.cpp#L357-L374

This initialization is the default, and `LIBCUDF_JIT_ENABLED` cannot clear it.

```cpp
DEFAULT = INIT_JIT_CACHE,
```

https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/include/cudf/context.hpp#L24

```cpp
void initialize(init_flags flags = init_flags::DEFAULT);
```

https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/include/cudf/context.hpp#L73

```cpp
auto const use_jit = detail::get_bool_env_or("LIBCUDF_JIT_ENABLED", false);
```

https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/src/runtime/context.cpp#L226

```cpp
flags = flags | (use_jit ? init_flags::INIT_JIT_CACHE : init_flags::NONE);
```

https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/src/runtime/context.cpp#L240

i.e. when `use_jit` is false, `flags | NONE` is effectively a no-op and so, the default value (`INIT_JIT_CACHE`) remains.

Java `computeColumn` calls `cudf::compute_column`, which always consults `get_context()` (and thus runs this init) before choosing the interpreter:
https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/src/transform/compute_column.cu#L36

Contributor guide

Open the contributing guide

Research direction

Reproduce the regression with Repro.java, then inspect cpp/src/runtime/context.cpp and cpp/src/transform/compute_column.cu, focusing on context initialization and the interpreter path. Compare the 26.06.0 and 26.08.1 behavior described in the issue and verify that computeColumn with LIBCUDF_JIT_ENABLED unset or false no longer requires libnvrtc.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.