google-research / google-research/inksight

GPU inference fails on Blackwell (RTX 5080, sm_120): CUDA_ERROR_INVALID_HANDLE during SavedModel execution

Open
#28 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1k
Forks
76
PR merge metrics
No merged PRs in 30d

Description

## Summary

I'm trying to run InkSight inference on an RTX 5080 Laptop (Blackwell, sm_120) inside Docker.
The service starts correctly, model loads, and TensorFlow can run basic GPU ops.
However, real InkSight inference fails on GPU with runtime errors (notably `CUDA_ERROR_INVALID_HANDLE`), while CPU execution works reliably.

This looks like a model-graph/runtime compatibility issue on this hardware/software stack, not a Docker mounting or basic GPU visibility issue.

---

## Environment

- GPU: NVIDIA GeForce RTX 5080 Laptop (compute capability 12.0 / sm_120)
- OS: Windows 11 (Docker Desktop)
- Container base: `nvcr.io/nvidia/tensorflow:25.02-tf2-py3`
- TensorFlow: `2.17.0+nv25.2`
- Python: `3.12.3`
- `tensorflow-text`: `2.18.1` (cp312 wheel)
- Model: `Derendering/InkSight-Small-p` (local mount)

Docker/runtime configuration:
- GPU device request enabled
- model mounted at `/data/models/inksight`
- `shm_size: 1gb`
- `ulimits`: memlock=-1, stack=67108864

---

## Expected behavior

`/vectorize` runs on GPU and returns SVG output reliably.

## Actual behavior

- `/health` is OK
- model loads
- basic TensorFlow GPU operations succeed
- InkSight inference graph fails on GPU with errors like:
- `CUDA_ERROR_INVALID_HANDLE`
- `cuLaunchKernel ... failed`
- failures around `Cast` / `StatefulPartitionedCall`
- CPU inference works and returns valid SVG

---

## Key error snippets

```text
INTERNAL: 'cuLaunchKernel(...)' failed with 'CUDA_ERROR_INVALID_HANDLE'
...
Detected at node Cast / StatefulPartitionedCall
```

And when trying a naive runtime switch to CPU (without reloading model):

```text
Trying to access resource ... located in device GPU:0 from device CPU:0
```

---

## What I already tried

### 1) Building `tensorflow-text` 2.17 from source inside NGC (instead of wheel)

Attempted to avoid potential wheel mismatch, but hit multiple blockers:
- Bazel WORKSPACE/repo issues related to NGC TF metadata (`@tf_wheel_version_suffix` / prepare script path)
- linker/version-script failures in pybind targets
- BUILD dependency drift (Abseil log deps)
- occasional Bazel external download/network failures

### 2) Current pragmatic stack

- Keep NGC TF 2.17.0+nv25.2
- Install `tensorflow-text==2.18.1` cp312 wheel
- Load model with `tf.saved_model.load()`

Result:
- stable service startup
- stable CPU inference
- unstable GPU inference on Blackwell

### 3) Fallback logic

- Direct `with tf.device(/CPU:0)` retry after GPU failure is not sufficient (device-bound variables)
- Reloading the SavedModel on CPU fixes fallback execution
- Current production workaround is CPU-preferred on sm_120 to avoid first-request GPU crash

---

## Ask

Could you suggest the recommended way to make InkSight inference stable on Blackwell GPU (sm_120) for this stack, or confirm if this requires an upstream fix?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.