NVENC nvEncInitializeEncoder returns NV_ENC_ERR_OUT_OF_MEMORY (10) on RTX PRO 6000 Blackwell Server Edition in SPT CC; NVDEC works;

NVIDIA Open GPU Kernel Modules Version

Driver Version: 595.84 CUDA Version (nvidia-smi): 13.2

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

OS: Ubuntu 24.04 (kernel: 6.8.0-90-generic)

Kernel Release

OS: Ubuntu 24.04 (kernel: 6.8.0-90-generic)

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

RTX pro 6000

Describe the bug

Environment

GPU: 1× NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), passthrough to one CVM

GPU CC: SPT (CC State = ON). Not Hopper PPCIe. Not Workstation / Max-Q SKU.

CPU TEE: Intel TDX guest

OS: Ubuntu 24.04 (kernel: 6.8.0-90-generic)

nvidia-smi: 595.84

Driver Version: 595.84

CUDA Version (nvidia-smi): 13.2

Video Codec SDK: 13.0.37 (official AppDec / AppEncCuda)

FFmpeg: libavcodec 60.31.102 (Ubuntu package; h264_nvenc present)

What works

AppDec (NVDEC) H.264 → NV12 succeeds (example: 300 frames, coded 1920×1088). Session init ~2.1 s.

nvEncOpenEncodeSessionEx succeeds. The sample prints the GPU as RTX PRO 6000 Blackwell Server Edition.

What fails (first encode session; card mostly idle)

Official sample AppEncCuda, after a successful OpenEncodeSessionEx:nvEncInitializeEncoder → NV_ENC_ERR_OUT_OF_MEMORY (10)Retry with -preset p1 -tuninginfo lowlatency -bf 0 -if nv12 -s 1920x1088: still error 10.Failure is before reading YUV / encoding any frame.

Independent client (no SDK sample, no YUV file):ffmpeg -hide_banner -y -f lavfi -i testsrc=size=1920x1080:rate=30 -t 2-c:v h264_nvenc -preset p1 -tune ll /tmp/nvenc.mp4[h264_nvenc] InitializeEncoder failed: out of memory (10)Follow-on messages (“maybe incorrect parameters”, “Cannot allocate memory”,“Conversion failed!”) are cascading; the root is error 10 at InitializeEncoder.

Questions (product statements, please)

Is NVENC supported on RTX PRO 6000 Blackwell Server Edition in SPT CC + Intel TDXwith driver 595.84 / CUDA 13.2?

If supported: is error 10 at nvEncInitializeEncoder a known bug? Fixed in whichdriver / VBIOS?

If not supported: which official document states that? Should playback encodebe done outside the TEE (CPU encode or non-CC GPU)?

On this SKU, is a CC-off (plaintext) control expected to succeed?

Already ruled out

No NVENC silicon. That is H200: OpenEncodeSessionEx error 2 = UNSUPPORTED_DEVICE.Here OpenEncodeSessionEx succeeds; only InitializeEncoder fails.

AppEncCuda-only CUDA buffer registration (ffmpeg fails at the same API).

Input YUV / resolution (lavfi testsrc 1920×1080; also retried 1920×1088 NV12).

Typical GeForce session-cap or 1–2 GB vGPU OOM: this is the first session on 96 GB.

To Reproduce

~/Video_Codec_SDK_13.0.37/Samples/build/AppEncode/AppEncCuda#ec SDK 13.0.37/Samples/build/AppDecode/AppDec/dec.yuv-Snfo lowlatency -bf 0-o /tmp/out.h264 -gpu 0

Bug Incidence

Always

nvidia-bug-report.log.gz

root@tdx-guest:~/Video_Codec_SDK_13.0.37/Samples/build/AppEncode/AppEncCuda#ec SDK 13.0.37/Samples/build/AppDecode/AppDec/dec.yuv-Snfo lowlatency -bf 0-o /tmp/out.h264 -gpu 0GPU in use: NVIDIA RTX PRO 6000 Blackwell Server Edition[INF0 ][08:53:10] Encoding Parameters:
1920x1088 -if nv12
codec
h264
preset
p1
tuningInfo
lowlatency(default)
profilechroma
yuv420
bitdepth
8
/AppEncCuda
-i /root/Video_Cod
-codec h264
-preset p1 -tuningu
rc
fps
gop
bf
multipass
sizebttrate
maxbitratevbvbufsizevbvinit
cbr
30/1
INF
0
0
1920x1088
0
0
0
0
disableddisableddisabled
temporalag
lookahead
cq
0
P,B,I=0,0,0P,B,I=0,0,0P,B,I=0,0,0CreateEncoder : m nvenc.nvEncInitializeEncoder(m hEncoder, &m_ initializeParamsideo_ Codec SDK_ 13.0.37/Samples/NvCodec/NvEncoder/NvEncoder.cpp:330
qmin
qmaxinitqp
) returned error 10 at /root/V

More Info

No response

Start by reproducing the failure with the mentioned Video Codec SDK 13.0.37 AppEncCuda sample and the FFmpeg h264_nvenc command. Inspect NvEncoder.cpp:330, where nvEncInitializeEncoder returns error 10, and compare it with the successful AppDec path. Done means determining whether this configuration is supported and documenting a confirmed driver, VBIOS, or environment cause for the initialization failure.