Comfy-Org / Comfy-Org/Nvidia_RTX_Nodes_ComfyUI

NvVFX SDK does not support multiple effect types across GPUs simultaneously

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
620
Forks
38
PR merge metrics
No merged PRs in 30d

Description

## Summary

The NVIDIA Video Effects (NvVFX) SDK has a **design limitation** where GPU selection (`NvVFX_SetS32(NULL, NVVFX_GPU, whichGPU)`) is a **global, one-time setting** — not per-instance. Running multiple effect types (e.g., denoise + super-resolution + deblur) simultaneously across multiple GPUs causes `NvVFX_Run` to fail with error code **-7** ("invalid parameter value") or **-1** ("unspecified error") on all GPUs except the first.

This affects any multi-GPU implementation that creates denoise, SR, and deblur instances on different devices concurrently.

## Environment

- **NvVFX SDK**: v1.2.0 (`nvvfx` Python package)
- **GPUs**: 1× RTX 4090 (SM 8.9) + 3× RTX 3090 (SM 8.6)
- **Driver**: 591.86
- **OS**: Windows 11
- **PyTorch**: 2.10.0+cu130

## Reproduction

### Works ✅ — Single effect type across all GPUs concurrently

```python
import torch, nvvfx, concurrent.futures
from nvvfx.effects import QualityLevel

# SR-only: all 4 GPUs succeed
instances = {}
for gpu_id in range(4):
with torch.cuda.device(gpu_id):
sr = nvvfx.VideoSuperRes(QualityLevel.HIGH, device=gpu_id)
sr.output_width, sr.output_height = 856, 480
sr.load()
instances[gpu_id] = sr

def worker(gpu_id):
frame = torch.rand(3, 240, 428, device=f"cuda:{gpu_id}", dtype=torch.float32)
torch.cuda.set_device(gpu_id)
out = instances[gpu_id].run(frame)
return torch.from_dlpack(out.image).clone()

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as ex:
futures = {ex.submit(worker, g): g for g in range(4)}
for f in concurrent.futures.as_completed(futures):
print(f"GPU {futures[f]}: OK") # All 4 succeed
```

### Works ✅ — Full pipeline on a single GPU

```python
# denoise → SR → deblur on one GPU: works perfectly
dn = nvvfx.VideoSuperRes(QualityLevel.DENOISE_MEDIUM, device=1)
sr = nvvfx.VideoSuperRes(QualityLevel.HIGH, device=1)
db = nvvfx.VideoSuperRes(QualityLevel.DEBLUR_MEDIUM, device=1)
# ... load all, run sequentially — all succeed
```

### Fails ❌ — Multiple effect types across GPUs concurrently

```python
# Create denoise + SR + deblur on each GPU, run concurrently
instances = {}
for gpu_id in range(4):
with torch.cuda.device(gpu_id):
dn = nvvfx.VideoSuperRes(QualityLevel.DENOISE_MEDIUM, device=gpu_id)
dn.output_width, dn.output_height = 432, 240
dn.load()
sr = nvvfx.VideoSuperRes(QualityLevel.HIGH, device=gpu_id)
sr.output_width, sr.output_height = 856, 480
sr.load()
db = nvvfx.VideoSuperRes(QualityLevel.DEBLUR_MEDIUM, device=gpu_id)
db.output_width, db.output_height = 856, 480
db.load()
instances[gpu_id] = (dn, sr, db)

# Run pipeline on all GPUs concurrently → GPU 0 succeeds, GPUs 1-3 fail
# Error: "NvVFX_Run failed: An invalid parameter value (code -7)"
# or: "An otherwise unspecified error (code -1)"
```

### Also fails ❌ — Even with serialized `run()` calls (lock)

```python
run_lock = threading.Lock()
# Same setup, but wrap each run() in: with run_lock: inst.run(frame)
# Still fails — proving it's not a race condition during execution
```

### Also fails ❌ — Even with identical GPUs (3× RTX 3090 only)

```python
# 3× RTX 3090, no 4090 involved:
# GPU 1: OK, GPUs 2-3: FAIL
# Only the first GPU to start processing succeeds
```

## Root Cause

The [NvVFX SDK Programming Guide](https://docs.nvidia.com/deeplearning/maxine/vfx-sdk-programming-guide/index.html) states:

> `NvVFX_SetS32(NULL, NVVFX_GPU, whichGPU)` **is intended to be called only once** for the Video Effects SDK before any effects are created.

> Images that are allocated on one GPU cannot be transparently passed to another GPU, so **you must ensure that the same GPU is used for all video effects.**

GPU selection is a **global SDK-wide state**, not per-instance. When creating instances on different GPUs, each creation overwrites the global GPU setting. The TensorRT engines for different effect types end up internally bound to the wrong GPU context, causing `NvVFX_Run` to fail.

An [unanswered forum post](https://forums.developer.nvidia.com/t/maxine-videofx-on-multiple-cuda-context/228998) from another developer asked the same question about multiple CUDA contexts — no response from NVIDIA.

## Workaround: Phased Processing

Process **one effect type at a time** across all GPUs — create instances, process all frames, close instances, then move to the next effect type:

```
Phase 1: Create denoise on all GPUs → process all frames → close all denoise
Phase 2: Create SR on all GPUs → process all frames → close all SR
Phase 3: Create deblur on all GPUs → process all frames → close all deblur
```

This ensures only one effect type exists globally at any time, matching the SDK's single-GPU design. Intermediate results are stored in CPU tensors between phases.

**Performance cost**: ~1-2s fixed overhead per run (3× instance creation/TensorRT engine loading instead of 1×), plus 2× additional CPU↔GPU data transfer for intermediate frames. For small frames (e.g., 428×240) and 150+ frame batches, the overhead is modest (~5-10% of total processing time).

## Tested Confirmation

| Scenario | Result |
|---|---|
| SR-only, 4 GPUs concurrent | ✅ All GPUs OK |
| Deblur-only, 4 GPUs concurrent | ✅ All GPUs OK |
| Denoise-only, 4 GPUs concurrent | ✅ All GPUs OK |
| Denoise+SR, 4 GPUs concurrent | ❌ Only first GPU OK |
| Full pipeline, single GPU | ✅ OK |
| Full pipeline, phased approach | ✅ All GPUs OK |
| Full pipeline with lock around run() | ❌ Still fails |

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.