facebookresearch / facebookresearch/sam3

Feature request: batched & multi-GPU inference support for SAM3 Agent

Open
#309 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.7k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

### Feature request: batched + multi-GPU inference for SAM3 Agent

Hi, thanks for releasing SAM 3 and the Agent pipeline!

I’m currently using the Agent interface (e.g. `sam3/agent/inference.py` + `run_single_image_inference`) with a vLLM backend to run text-driven segmentation on a large image dataset on a multi-GPU server.

It would be very helpful to have:

1. **Batched Agent inference (single GPU)**
- A function like `run_batch_inference(...)` that takes a list of `(image_path, text_prompt)` pairs and runs the Agent in batch, reusing the same SAM3 model and LLM client.
- Output format could stay the same as the current per-image `*_pred.json` files.

2. **Multi-GPU inference recipe for the Agent**
- An example script (e.g. `examples/agent_multi_gpu.py`) using `torchrun` or similar to:
- Split a list of images across multiple GPUs.
- Run the Agent on each subset.
- Avoid collisions in `sam_out/` and `*_pred.json` outputs.

This would make it much easier to scale Agent-based experiments to 10k+ images on machines with 4–8 GPUs.

**Environment (example)**
- Linux, Python 3.x, PyTorch 2.x
- GPUs: multi-GPU server (e.g. 8 × A800)
- SAM3: latest from GitHub
- LLM backend: vLLM + Qwen/Qwen3-VL-32B-Instruct

If there is already a recommended way to do batched or multi-GPU Agent inference, pointers would be greatly appreciated. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with sam3/agent/inference.py and run_single_image_inference to understand the current per-image Agent flow and output files. Then review the requested examples/agent_multi_gpu.py approach using torchrun, including image partitioning and output collision avoidance. Done means batched inference and a multi-GPU recipe work for the described image and prompt inputs while preserving the existing prediction output format.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.