google-research / google-research/inksight
GPU inference fails on Blackwell (RTX 5080, sm_120): CUDA_ERROR_INVALID_HANDLE during SavedModel execution
- Dominant language
- Jupyter Notebook
- Stars
- 1k
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
I'm trying to run InkSight inference on an RTX 5080 Laptop (Blackwell, sm_120) inside Docker.
The service starts correctly, model loads, and TensorFlow can run basic GPU ops.
However, real InkSight inference fails on GPU with runtime errors (notably `CUDA_ERROR_INVALID_HANDLE`), while CPU execution works reliably.
This looks like a model-graph/runtime compatibility issue on this hardware/software stack, not a Docker mounting or basic GPU visibility issue.
---
## Environment
- GPU: NVIDIA GeForce RTX 5080 Laptop (compute capability 12.0 / sm_120)
- OS: Windows 11 (Docker Desktop)
- Container base: `nvcr.io/nvidia/tensorflow:25.02-tf2-py3`
- TensorFlow: `2.17.0+nv25.2`
- Python: `3.12.3`
- `tensorflow-text`: `2.18.1` (cp312 wheel)
- Model: `Derendering/InkSight-Small-p` (local mount)
Docker/runtime configuration:
- GPU device request enabled
- model mounted at `/data/models/inksight`
- `shm_size: 1gb`
- `ulimits`: memlock=-1, stack=67108864
---
## Expected behavior
`/vectorize` runs on GPU and returns SVG output reliably.
## Actual behavior
- `/health` is OK
- model loads
- basic TensorFlow GPU operations succeed
- InkSight inference graph fails on GPU with errors like:
- `CUDA_ERROR_INVALID_HANDLE`
- `cuLaunchKernel ... failed`
- failures around `Cast` / `StatefulPartitionedCall`
- CPU inference works and returns valid SVG
---
## Key error snippets
```text
INTERNAL: 'cuLaunchKernel(...)' failed with 'CUDA_ERROR_INVALID_HANDLE'
...
Detected at node Cast / StatefulPartitionedCall
```
And when trying a naive runtime switch to CPU (without reloading model):
```text
Trying to access resource ... located in device GPU:0 from device CPU:0
```
---
## What I already tried
### 1) Building `tensorflow-text` 2.17 from source inside NGC (instead of wheel)
Attempted to avoid potential wheel mismatch, but hit multiple blockers:
- Bazel WORKSPACE/repo issues related to NGC TF metadata (`@tf_wheel_version_suffix` / prepare script path)
- linker/version-script failures in pybind targets
- BUILD dependency drift (Abseil log deps)
- occasional Bazel external download/network failures
### 2) Current pragmatic stack
- Keep NGC TF 2.17.0+nv25.2
- Install `tensorflow-text==2.18.1` cp312 wheel
- Load model with `tf.saved_model.load()`
Result:
- stable service startup
- stable CPU inference
- unstable GPU inference on Blackwell
### 3) Fallback logic
- Direct `with tf.device(/CPU:0)` retry after GPU failure is not sufficient (device-bound variables)
- Reloading the SavedModel on CPU fixes fallback execution
- Current production workaround is CPU-preferred on sm_120 to avoid first-request GPU crash
---
## Ask
Could you suggest the recommended way to make InkSight inference stable on Blackwell GPU (sm_120) for this stack, or confirm if this requires an upstream fix?
Contributor guide
Assessment
This issue has not been assessed yet.