Performance Bottleneck in SAM2 Multi-Instance Propagation (Parallel Execution)

Hello all,

I’m currently working on the SAM2 predictor, which performs mask propagation over a set of images using bounding boxes. My use case involves multiple box initializations, each triggering an independent propagation sequence.

To speed things up, I’ve initialized two separate instances of the predictor class and run them in parallel using multiprocessing. The hardware setup is as follows:

GPU: NVIDIA A5000 (24 GB VRAM)

CPU: 32-core

RAM: 128 GB

Here’s the parallel execution logic in simplified form:

```
predictor_1 = build_sam2_video_predictor(CONFIG_PATH, CHECKPOINT_PATH)
Auto_annotation_video_object_1 = Auto_annotation_video(predictor_1)
results_1 = Auto_annotation_video_object_1.auto_propagate_video(frames_dir, 58, predictor_1, bbox_1)

predictor_2 = build_sam2_video_predictor(CONFIG_PATH, CHECKPOINT_PATH)
Auto_annotation_video_object_2 = Auto_annotation_video(predictor_2)
results_2 = Auto_annotation_video_object_2.auto_propagate_video(frames_dir, 58, predictor_2, bbox_2)
```
When run sequentially:

One propagation (31 frames) takes ~1m 14s

When both predictors are run in parallel:

Both propagations take ~2m+, effectively no speed-up, despite:

Sufficient GPU headroom (16/24 GB VRAM used)

Adequate CPU and RAM

Observations
GPU utilization remains under 100% and does not scale linearly with multiple instances.

Running concurrent SAM2 jobs shows signs of GPU serialization (likely due to shared CUDA context or memory bandwidth contention).

CPU utilization also stays moderate, no evidence of CPU-side bottleneck.

Attempts Made
Ran with:

multiprocessing.Process

concurrent.futures.ProcessPoolExecutor

concurrent.futures.ThreadPoolExecutor

Pinned each process to separate CPU cores to avoid contention.

Checked for shared GPU context interference or session-level locks.

Question
Has anyone successfully vertically scaled multiple SAM2 predictors on a single GPU?
How can I architect this better to maximize GPU concurrency and minimize propagation time?

Any insights on:

CUDA stream parallelism?

Separate process group initialization for GPU context isolation?

Memory pinning, or per-process VRAM constraints?

Would appreciate any guidance from those who have optimized similar inference pipelines with multiple concurrent deep learning models.