Kernel slab leak: NvKmsKapiSemaphoreSurfaceCallback (semaphore-surface waiter) never freed under Wayland explicit sync — 595.84 & 610.43.02

Summary

Under a steady Wayland explicit-sync rendering workload, the open kernel modules
leak 64-byte struct NvKmsKapiSemaphoreSurfaceCallback objects
(nvKmsKapiCalloc → nvInternalAlloc → nvkms_alloc) at a high, unbounded
rate. The objects are allocated in the semaphore-surface waiter registration
path (__nv_drm_semsurf_ctx_reg_callbacks, kernel-open/nvidia-drm/nvidia-drm-fence.c)
and are never freed. Only a reboot reclaims the memory. Reproduced on both
595.84 and the latest packaged open driver 610.43.02.

Environment

• GPU: RTX 5070 Ti (GB203, Blackwell)
• Driver: NVIDIA open kernel modules 595.84 and 610.43.02 (both affected)
• OS/kernel: Ubuntu 26.04, kernel 7.0.0-30-generic
• Compositor: headless sway 1.11 / wlroots 0.19.1 + a Vulkan app at 30–60 fps
  (8 instances during measurement); Wayland explicit sync

Impact

• Leak rate ~3,600 objects/second (64-byte slab). SUnreclaim climbs
  continuously; observed ~605 million live objects in one kmalloc-64 cache and
  SUnreclaim 38.5 GiB after ~22 h. A 59 GiB machine becomes unusable in ~2
  days; only a reboot recovers.
• Local denial of service (CWE-401 → CWE-400). Not remote, not privilege
  escalation, no data exposure.

Evidence (bpftrace on the running modules)

Per-pointer 64-byte alloc/free matching, 60 s window under the workload, on
610.43.02:

call site	allocated	freed	leaked
nvkms_alloc	232,139	14,209	~217,930 / min (~3,630/s)
nv_drm_calloc	344,633	344,633	0 (balanced)
drm_syncobj_create	80,671	80,669	0 (balanced)

SUnreclaim rose 466,792 → 566,688 kB (+~100 MB) within a few minutes and keeps
climbing. nv_drm_calloc (the nvidia-drm partner struct allocated in the same
function) is perfectly balanced right next to the leaking nvkms_alloc, which
isolates the leak to the nvidia-modeset callback object.

Callback lifecycle rates (20 s window): __nv_drm_semsurf_ctx_reg_callbacks
77,603 calls vs __nv_drm_semsurf_ctx_callback (the callback actually firing,
which is what frees the nvkms cb in SemaphoreSurfaceKapiCallback,
nvkms-kapi-sync.c) only 3,381 — a ~23:1 registered-vs-fired ratio, identical on
595.84 and 610.43.02.

Analysis

A struct NvKmsKapiSemaphoreSurfaceCallback (cb) is allocated per waiter
registration in nvKmsKapiRegisterSemaphoreSurfaceCallback(). It is freed only
when (a) the callback fires (SemaphoreSurfaceKapiCallback → nvKmsKapiFree),
or (b) nvKmsKapiUnregisterSemaphoreSurfaceCallback() returns success. Under
this workload the vast majority of registered waiters neither fire nor are
successfully unregistered: cb frees exactly track the fire count (~5k/20s),
and unregister frees are ~0.

The nvidia-drm side keeps a single in-flight callback slot per fence context
(ctx->callback) and, when a callback cannot be unregistered, assumes it "is
already running and will free itself" (__nv_drm_semsurf_ctx_store_callback
and __nv_drm_semsurf_ctx_reg_callbacks in nvidia-drm-fence.c). At high
fence throughput that assumption does not hold for the bulk of registrations, so
cb is neither fired nor unregistered and is orphaned in RM's per-value
listener list. nvidia-drm-fence.c is byte-identical on current main, so the
path is unchanged upstream.

Note (to save triage time): an earlier candidate — freeing cb on the
NVOS_STATUS_ERROR_ALREADY_SIGNALLED return of
nvKmsKapiRegisterSemaphoreSurfaceCallback() — was tested with a built/signed
patch and did not reduce the leak; that branch is rarely taken (nvkms cb
allocations < reg_callbacks calls, so the ALREADY_SIGNALLED retry loop is not
the source). Please do not treat that as the fix.

Additional diagnostics (to narrow root cause)

Object fingerprint. The leaking allocations fall in the [32, 64)
requested-bytes bucket (measured via tracepoint:kmem:kmalloc bytes_req),
consistent with sizeof(struct NvKmsKapiSemaphoreSurfaceCallback) plus the
nvInternalAlloc header — i.e. the cb, not some other kmalloc-64 tenant.

Allocation stack (frame-pointer unwind):

__kmalloc_noprof
nvkms_alloc
nvInternalAlloc
  (inlined nvKmsKapiCalloc, from nvKmsKapiRegisterSemaphoreSurfaceCallback,
   driven by nvidia-drm __nv_drm_semsurf_ctx_reg_callbacks →
   nvKms->registerSemaphoreSurfaceCallback)

The stack cannot be unwound deeper than nvInternalAlloc: the nvidia-modeset
module is built without frame pointers and its internal symbols
(nvkms_alloc, nvKmsKapi*, SemaphoreSurfaceKapiCallback) are not in
available_filter_functions, so they are neither kprobe-traceable nor
BPF-unwindable (nvidia-drm symbols, by contrast, are traceable). If nvidia-modeset
shipped ftrace/frame-pointer support this class of leak would be far easier for
users to attribute.

What we ruled out (so you don't have to):

• The RM notify/delivery path is correct for values that are actually reached:
  _semsurfEventCallback moves every listener with semValue >= value (jumps
  included) into the notify list and _semsurfNotifyCompleted delivers via
  osEventNotification then auto-removes it. The event-index match also checks
  out — registerEventNotification masks NV01_EVENT_WITHOUT_EVENT_DATA
  (event_notification.c), so the registered NotifyIndex (0) equals the
  dispatch notifyIndex (0); the kernel callback is invoked.
• So the leak is not a delivery bug. The orphaned waiters are those whose
  wait value is never reached on the semaphore (the nvidia-drm side polls
  fence seqnos in __nv_drm_semsurf_ctx_process_completed and completes /
  supersedes fences directly), and they are never unregistered:
  _semsurfDelWaiter returns not-found and cb frees via unregister are ~0
  (measured). The single-slot ctx->callback + "unregister failed ⇒ it's
  running, it will free itself" assumption strands them permanently in RM's
  per-value listener list.

Possible fix direction (tentative, not verified — a UAF-safe change is
needed since a stored handle may still fire): have nvidia-drm track all
outstanding cb handles for a context (rather than a single slot) and
guarantee each is unregistered/reclaimed when it is superseded or when the
context stops tracking it, instead of assuming a non-unregisterable callback
will self-free.

Secondary, unconfirmed: os_alloc_mem shows a small net 64-byte imbalance
under the same workload (155 alloc / 27 freed in 60 s) — possibly related RM
bookkeeping, not root-caused; flagging in case it shares a cause.

Reproducer

Headless sway (WLR_BACKENDS=headless, WLR_RENDER_DRM_DEVICE=<nvidia render node>) plus any Vulkan client rendering at ~60 fps; watch
grep SUnreclaim /proc/meminfo climb ~60–100 kB/s per instance. Steady
rendering is sufficient; process churn is not required. The bpftrace scripts
used for all measurements above (per-pointer 64-byte alloc/free matching,
per-caller attribution, callback-lifecycle counters) are available on request —
happy to attach them or test candidate patches on the affected hardware.

Start with kernel-open/nvidia-drm/nvidia-drm-fence.c, especially __nv_drm_semsurf_ctx_reg_callbacks, __nv_drm_semsurf_ctx_store_callback, and __nv_drm_semsurf_ctx_process_completed, then trace the related nvkms callback registration and unregister paths. Reproduce with headless sway and a steady Vulkan workload while monitoring SUnreclaim and callback allocations. Done means superseded waiters are safely reclaimed without use-after-free and the unbounded leak no longer occurs.