Counter Collection Unit (CCU) support for GH200

NVIDIA Open GPU Kernel Modules Version

570.172.08

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 24.04.2 LTS

Kernel Release

Linux 6.8.0-1030-nvidia-64k #33-Ubuntu SMP PREEMPT_DYNAMIC

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA GH200 480GB (UUID: GPU-371bf4a2-5abb-1598-c28a-74ef62ae1f0b)

Describe the bug

The CCU (Counter Collection Unit) buffer allocation on NVIDIA GH200 (480GB Grace Hopper Superchip) fails due to improper initialization of the CCU engine in the production driver (Version 570.172.08). The expected CCU buffer size is 0 bytes, which suggests that the CCU shared buffer is not being properly allocated during the GPU engine setup.

To Reproduce

Install NVIDIA Kernel Modules:

Clone the open GPU kernel modules repository:

git clone https://github.com/NVIDIA/open-gpu-kernel-modules.git
cd open-gpu-kernel-modules

Build the kernel modules:

make modules -j$(nproc)

Install the modules:

sudo make modules_install -j$(nproc)

Install NVIDIA-SMI Without Kernel Modules:

Download the NVIDIA driver:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/NVIDIA-Linux-x86_64-570.172.08.run

Install the driver without kernel modules:

sudo sh ./NVIDIA-Linux-x86_64-570.172.08.run --no-kernel-modules

Restart the system to apply changes:

sudo reboot

Verify Installation:

Check the GPU status using nvidia-smi:

nvidia-smi

Access CCU Functionality:

ls /proc/driver/nvidia/ccu

#Directory not found.

#Tried initiazing and reading from CCU

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

Issue: CCU Shared Buffer Allocation Failure

The CCU shared buffer is never allocated because the function kccuStateLoad() is not being called during GPU initialization. The call chain fails at:

kccuapiCtrlCmdSubscribe_IMPL()
↓
kccuMemDescGetForShrBufId(pGpu, pKernelCcu, CCU_DEV_SHRBUF_ID, &pMemDesc)
↓
pKernelCcu->pMemDesc[CCU_DEV_SHRBUF_ID] = NULL
↓
Returns NV_ERR_NO_MEMORY (0x1f)

---

Fixes Attempted:

1. Modified /etc/modprobe.d/nvidia.conf

• Parameters Tried:
  • EnableStreamMemOPs=1
  • EnableCCU=1
  • RMEnableCCU=1
  • EnableHWPM=1
• Result: No change in CCU buffer allocation

---

2. NVIDIA Registry Modification

• Method: Direct writes to /proc/driver/nvidia/registry
• Registry Keys Attempted:
  • EnableStreamMemOPs=1
  • EnableGpuFirmware=1
  • EnableCCU=1
  • RMEnableCCU=1
  • EnableHWPM=1
  • EnableCounterCollection=1
  • CCUEnabled=1
  • EnablePerfMon=1
• Result: No change in CCU buffer allocation

---

3. Module Reload and System Restart

• Method: Complete driver reload and system reboot
• Commands Used:

  sudo modprobe -r nvidia
  sudo modprobe nvidia
  sudo reboot
  
  
  

More Info

No response

Start by tracing kccuapiCtrlCmdSubscribe_IMPL() through kccuMemDescGetForShrBufId() and the missing kccuStateLoad() call during GPU initialization. Reproduce on the stated GH200 environment and inspect the open kernel module initialization path; done means the CCU shared buffer is allocated and /proc/driver/nvidia/ccu is available.