Host memory grows ~6.8 KB per inference on aarch64 + CUDA (not on x86_64, not on CPU)

Host memory grows ~6.8 KB per inference on aarch64 + CUDA (not on x86_64, not on CPU)

Summary

Running WhisperModel.transcribe() in a loop with a resident model and fixed-shape
input leaks about 6.8 KB of host RSS per call on aarch64 + CUDA. The growth is
strictly linear and is not recovered by gc.collect() or malloc_trim(0).

The same test shows no growth on x86_64 + CUDA and no growth on aarch64 + CPU,
using the same CTranslate2 version (4.8.1) and the same script.

At a realistic streaming-ASR rate (one 5-second chunk every 5 seconds) this is
~117 MB/day, which we confirmed in a 24-hour run.

This is host memory (RSS), not GPU memory — which distinguishes it from the
previously reported GPU-side growth in #1488 and SYSTRAN/faster-whisper#660.

Environment matrix

All rows: CTranslate2 4.8.1, faster-whisper 1.2.1, model Systran/faster-whisper-small,
compute_type=float16 (CPU row: float32), beam_size=1, no VAD,
input = twelve fixed 5-second 16 kHz mono WAV files cycled in order.

#	CPU arch	Device	CUDA	CT2 build	Python	OS	Result
1	x86_64 (i9-14900KF, RTX 4090)	CUDA	12.4	official PyPI wheel	3.10	Ubuntu 22.04	no growth
2	aarch64 (Jetson AGX Orin)	CPU	–	official PyPI wheel	3.12	Ubuntu 24.04	no growth
3	aarch64 (Jetson AGX Orin, sm_87)	CUDA	13.2 (JetPack 7.2)	built from source	3.12	Ubuntu 24.04	+6.8 KB/call
4	aarch64 (NVIDIA GB10, sm_121)	CUDA	13.0 (plain Ubuntu)	built from source	3.12	Ubuntu 24.04	+6.8 KB/call

Rows 3 and 4 are different machines, different GPUs, different CUDA versions, and were
built independently — yet produce the same per-call figure.

Measurements

Every 250 iterations we call gc.collect() then malloc_trim(0), and record RSS after
each. The table shows RSS after malloc_trim(0) (i.e. memory that cannot be reclaimed).

iterations	Jetson AGX Orin (row 3)	GB10 (row 4)
250	1533.3 MB	808.4 MB
500	1594.9 MB	810.2 MB (+1.8)
750	1596.6 MB (+1.7)	812.0 MB (+1.8)
1000	1598.3 MB (+1.7)	813.6 MB (+1.6)
1250	1600.0 MB (+1.7)	815.2 MB (+1.6)

+1.6–1.8 MB per 250 calls = ~6.8 KB per call, on both machines, with no sign of
levelling off. (The larger jump before 500 iterations is the allocator pool filling up;
that part does saturate. The linear part after it does not.)

x86_64 (row 1) over the same 1250 iterations: -0.8 to +0.4 MB, i.e. flat.

What we ruled out

• Python-side accumulation — gc.collect() reclaims 0.0 MB and the number of
  objects tracked by gc grows by +5 over 1250 iterations, on all four rows.

• glibc arena retention — malloc_trim(0) does reclaim a steady amount every time
  (7.4–66.6 MB depending on the machine), but it does not change the slope.

• The CUDA allocator implementation — we ran row 4 again with
  CT2_CUDA_ALLOCATOR=cub_caching. The switch demonstrably took effect (the amount
  reclaimed by malloc_trim changed from 31.6 MB to 7.3 MB per call), but the slope was
  identical:

  iterations	cuda_malloc_async (default)	cub_caching
  250	+2.3 MB	+2.0 MB
  500	+4.1 MB	+4.1 MB
  750	+5.9 MB	+5.4 MB
  1000	+7.5 MB	+7.5 MB
  1250	+9.0 MB	+9.3 MB

  So this is not CudaAsyncAllocator / cudaMallocAsync, which is what
  SYSTRAN/faster-whisper#660 points at for a different (crash) symptom.

• A broken local build — rows 3 and 4 were configured and compiled separately, on
  different machines, for different compute capabilities, and agree to the same figure.

• Variable input shapes — every call uses one of the same twelve 5-second files, so
  shape-keyed caches cannot explain unbounded growth.

Reproduction

import ctypes, gc, sys, time
from faster_whisper import WhisperModel

libc = ctypes.CDLL("libc.so.6")

def rss_mb():
    for line in open("/proc/self/status"):
        if line.startswith("VmRSS:"):
            return int(line.split()[1]) / 1024

chunks = sys.argv[1:]                      # twelve 5-second 16 kHz mono WAV files
model = WhisperModel("small", device="cuda", compute_type="float16")
list(model.transcribe(chunks[0], language="ja", beam_size=1)[0])
gc.collect(); libc.malloc_trim(0)
base, base_obj = rss_mb(), len(gc.get_objects())
print(f"[ready] rss={base:.1f}MB obj={base_obj}")

for i in range(1, 1251):
    segments, _ = model.transcribe(chunks[i % len(chunks)], language="ja", beam_size=1)
    _ = "".join(s.text for s in segments)
    if i % 250 == 0:
        r0 = rss_mb(); gc.collect()
        r1 = rss_mb(); libc.malloc_trim(0)
        r2 = rss_mb()
        print(f"{i:5d}  rss={r0:7.1f}  after_gc={r1:7.1f}  after_trim={r2:7.1f}  "
              f"net={r2-base:+6.1f}MB  gc_objects={len(gc.get_objects())-base_obj:+d}")

Build configuration used for rows 3 and 4 (stock apart from the arch list):

-DWITH_CUDA=ON -DWITH_CUDNN=ON -DWITH_MKL=OFF -DWITH_OPENBLAS=OFF -DWITH_DNNL=OFF
-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DOPENMP_RUNTIME=COMP

Note for row 4: CMake 3.28's cuda_select_nvcc_arch_flags does not recognise sm_121,
so that call was replaced with an explicit
-gencode arch=compute_121,code=sm_121. Row 3 (sm_87) was built without any patch and
produces the same figure, so this workaround is not related to the leak.

Impact

For always-on speech recognition this is the difference between a process that runs
indefinitely and one that has to be restarted. On an 8 GB Jetson Orin Nano the practical
budget is about 20 days; we currently restart the recogniser process once a day as a
workaround (reloading the model takes ~6 s, so this is cheap — but it should not be
necessary).

Related

• #1488 — GPU memory growth on x86_64 with variable input sizes. Different symptom
  (device memory, not host RSS) and different trigger (variable shapes).
• SYSTRAN/faster-whisper#660 — several reports there point at
  ctranslate2::cuda::cudaasyncallocator::free(), but that is a crash signature; our
  cub_caching run above shows the steady host-side growth happens without that
  allocator.

Start with the provided Python reproduction using WhisperModel.transcribe and verify the linear host-RSS growth on aarch64 with CUDA. Inspect the CudaAsyncAllocator path and compare it with CT2_CUDA_ALLOCATOR=cub_caching, then trace the relevant C++ CUDA inference code. Done means the slope is eliminated for fixed-shape repeated inference while the reported CPU and x86_64 behavior remains unchanged.