[BUG] computeColumn in 26.08.1 fails without NVRTC
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Maven Central `ai.rapids:cudf:26.08.1` (`cuda13` classifier) throws an RTCX error on `CompiledExpression.computeColumn` when `libnvrtc.so.13` is not on the host. The same program succeeds with `ai.rapids:cudf:26.06.0` (`cuda13`).
`LIBCUDF_JIT_ENABLED` is unset (default `false`). This is the AST interpreter path, not an explicit JIT compile. Setting `LIBCUDF_JIT_ENABLED=false` does not avoid the failure.
```
ai.rapids.cudf.CudfException: RTCX failure at: /__w/cudf/cudf/.java-work/libcudf-build/_deps/rtcx-src/rtcx.cpp:285: Failed to load dynamic library `libnvrtc.so` (tried: libnvrtc.so.13)
at ai.rapids.cudf.ast.CompiledExpression.computeColumn(Native Method)
at ai.rapids.cudf.ast.CompiledExpression.computeColumn(CompiledExpression.java:77)
```
The classifier JAR does not ship NVRTC.
**Steps/Code to reproduce bug**
`Repro.java`:
```java
import ai.rapids.cudf.ColumnVector;
import ai.rapids.cudf.Table;
import ai.rapids.cudf.ast.ColumnReference;
import ai.rapids.cudf.ast.CompiledExpression;
public class Repro {
public static void main(String[] args) {
try (ColumnVector c = ColumnVector.fromInts(1, 2, 3);
Table t = new Table(c);
CompiledExpression e = new ColumnReference(0).compile();
ColumnVector r = e.computeColumn(t)) {
System.out.println("OK rows=" + r.getRowCount());
}
}
}
```
```bash
curl -fsSLO https://repo1.maven.org/maven2/ai/rapids/cudf/26.06.0/cudf-26.06.0-cuda13.jar
curl -fsSLO https://repo1.maven.org/maven2/ai/rapids/cudf/26.08.1/cudf-26.08.1-cuda13.jar
curl -fsSLO https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar
# 26.06.0 -> OK
docker run --rm --gpus all -v "$PWD":/workspace -w /workspace eclipse-temurin:17-jdk \
java -cp 'cudf-26.06.0-cuda13.jar:slf4j-api-1.7.30.jar' Repro.java
# 26.08.1 -> RTCX failure
docker run --rm --gpus all -v "$PWD":/workspace -w /workspace eclipse-temurin:17-jdk \
java -cp 'cudf-26.08.1-cuda13.jar:slf4j-api-1.7.30.jar' Repro.java
```
26.06.0:
```
OK rows=3
```
26.08.1:
```
Exception in thread "main" ai.rapids.cudf.CudfException: RTCX failure at: /__w/cudf/cudf/.java-work/libcudf-build/_deps/rtcx-src/rtcx.cpp:285: Failed to load dynamic library `libnvrtc.so` (tried: libnvrtc.so.13)
at ai.rapids.cudf.ast.CompiledExpression.computeColumn(Native Method)
at ai.rapids.cudf.ast.CompiledExpression.computeColumn(CompiledExpression.java:77)
at Repro.main(Repro.java:11)
```
**Expected behavior**
With `LIBCUDF_JIT_ENABLED` unset/`false`, `computeColumn` should use the AST interpreter and not require `libnvrtc`, matching 26.06.0.
**Environment overview (please complete the following information)**
- Environment location: Docker
- Method of cuDF install: Maven Central classifier JARs (`ai.rapids:cudf:26.06.0:cuda13` and `ai.rapids:cudf:26.08.1:cuda13`) plus declared `slf4j-api`
```bash
docker pull eclipse-temurin:17-jdk
docker run --rm --gpus all -v "$PWD":/workspace -w /workspace eclipse-temurin:17-jdk \
java -cp 'cudf-26.08.1-cuda13.jar:slf4j-api-1.7.30.jar' Repro.java
```
Host: x86_64 Linux, NVIDIA Container Toolkit (`--gpus all`). The JDK image has no CUDA toolkit and no `libnvrtc`.
**Environment details**
Not Applicable.
**Additional context**
In 26.08, libcudf replaced jitify with librtcx (https://github.com/rapidsai/cudf/pull/22654), and this update appears to have moved `libnvrtc` loading to the JIT cache initialization step (this previously happened during first kernel compilation).
In 26.06, cache initialization only built the jitify program cache
```cpp
void context::ensure_jit_cache_initialized()
{
std::call_once(_program_cache_init_flag,
[&]() { _program_cache = jit::program_cache::create(); });
}
```
https://github.com/NVIDIA/cudf/blob/v26.06.00/cpp/src/runtime/context.cpp#L27-L31
and did not load `libnvrtc`. My understanding is that jitify loads `libnvrtc` lazily when a kernel is actually compiled, so with `LIBCUDF_JIT_ENABLED` set to false, `libnvrtc` was never loaded.
In 26.08, the same function unconditionally initializes rtcx, and rtcx loads `libnvrtc` inside `initialize()`
```cpp
void context::ensure_jit_cache_initialized()
{
std::call_once(_jit_cache_init_flag, [&]() {
// make sure the required directories exist
std::filesystem::create_directories(_config.rtcx_cache_dir);
std::filesystem::create_directories(_config.jit_bundle_dir);
std::filesystem::create_directories(_config.jit_pch_dir);
std::filesystem::create_directories(_config.jit_tmp_dir);
rtcx::initialize();
...
```
https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/src/runtime/context.cpp#L65-L74
See `LibNVRTC::_load()` call in `rtcx::initialize()`:
https://github.com/rapidsai/librtcx/blob/efad266c1fd9de6d8486c6ba71bfa74df063eb1f/rtcx.cpp#L463-L474
`LibNVRTC::_load()` `dlopen`s `libnvrtc.so.`:
https://github.com/rapidsai/librtcx/blob/efad266c1fd9de6d8486c6ba71bfa74df063eb1f/rtcx.cpp#L357-L374
This initialization is the default, and `LIBCUDF_JIT_ENABLED` cannot clear it.
```cpp
DEFAULT = INIT_JIT_CACHE,
```
https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/include/cudf/context.hpp#L24
```cpp
void initialize(init_flags flags = init_flags::DEFAULT);
```
https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/include/cudf/context.hpp#L73
```cpp
auto const use_jit = detail::get_bool_env_or("LIBCUDF_JIT_ENABLED", false);
```
https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/src/runtime/context.cpp#L226
```cpp
flags = flags | (use_jit ? init_flags::INIT_JIT_CACHE : init_flags::NONE);
```
https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/src/runtime/context.cpp#L240
i.e. when `use_jit` is false, `flags | NONE` is effectively a no-op and so, the default value (`INIT_JIT_CACHE`) remains.
Java `computeColumn` calls `cudf::compute_column`, which always consults `get_context()` (and thus runs this init) before choosing the interpreter:
https://github.com/NVIDIA/cudf/blob/v26.08.01/cpp/src/transform/compute_column.cu#L36
Contributor guide
Research direction
Reproduce the regression with Repro.java, then inspect cpp/src/runtime/context.cpp and cpp/src/transform/compute_column.cu, focusing on context initialization and the interpreter path. Compare the 26.06.0 and 26.08.1 behavior described in the issue and verify that computeColumn with LIBCUDF_JIT_ENABLED unset or false no longer requires libnvrtc.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100