facebookresearch / facebookresearch/sam3
Add `max_generations` parameter to `run_single_image_inference()` to allow configurable agent iteration limit
- Dominant language
- Python
- Stars
- 11.7k
- Forks
- 1.8k
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
The SAM3 agent's `agent_inference()` function supports a `max_generations` parameter (default: 100) to limit the number of agent rounds. However, `run_single_image_inference()` does not expose this parameter, making it impossible for downstream users to configure a lower limit without modifying SAM3 source code.
This enhancement would enable users to set sensible early-stopping limits for use cases where the target segment may not exist in the image, preventing excessive LLM API calls and runtime.
---
### Problem Statement
When using the SAM3 agent via `run_single_image_inference()`, there is no way to control the maximum number of segmentation rounds the agent will attempt. The agent will continue iterating up to the hardcoded default of 100 generations (defined in `agent_inference()`).
**Use case:** A segmentation pipeline where the target object may not always be present or semi covered, the agent can waste a lot of time and API costs trying multiple prompt variations when a target simply doesn't exist. Therefore, I request a way to configure a lower threshold (e.g., 3-5 rounds) to fail fast.
---
### Proposed Solution
Add an optional `max_generations` parameter to `run_single_image_inference()` and pass it through to `agent_inference()`.
**Files to modify:**
- `sam3/agent/inference.py`
**Proposed changes:**
```python
def run_single_image_inference(
image_path,
text_prompt,
llm_config,
send_generate_request,
call_sam_service,
output_dir="agent_output",
debug=False,
max_generations=100, # NEW: Add parameter with default matching agent_inference
):
"""Run inference on a single image with provided prompt
Args:
...existing args...
max_generations: Maximum number of agent rounds to attempt (default: 100)
"""
# ...existing setup code...
agent_history, final_output_dict, rendered_final_output = agent_inference(
image_path,
text_prompt,
send_generate_request=send_generate_request,
call_sam_service=call_sam_service,
output_dir=output_dir,
debug=debug,
max_generations=max_generations, # NEW: Pass through parameter
)
# ...existing save code...
```
---
### Example Usage (After Change)
**Current (no control):**
```python
from sam3.agent.inference import run_single_image_inference
# Agent will try up to 100 rounds (hardcoded default)
output_image_path, output_json_path = run_single_image_inference(
image_path="house_exterior.jpg",
text_prompt="wooden deck, not concrete patio",
llm_config=config,
send_generate_request=my_llm_func,
call_sam_service=my_sam_func,
)
```
**Proposed (configurable):**
```python
from sam3.agent.inference import run_single_image_inference
# Agent will stop after 3 unsuccessful rounds
output_image_path, output_json_path = run_single_image_inference(
image_path="house_exterior.jpg",
text_prompt="wooden deck, not concrete patio",
llm_config=config,
send_generate_request=my_llm_func,
call_sam_service=my_sam_func,
max_generations=3, # NEW: User can specify limit
)
```
---
### Real-World Example
**Scenario:** Processing a batch of 1,000 real estate images for property analysis. Some images contain the target segment (e.g., "wooden deck"), some don't (e.g., houses without decks, obstructed views) and the rest has a mix wooden deck and other.
**Current behavior:**
- Agent tries up to 100 rounds per image
- For complicated images, e.g., without the target, this potentially returns ~100 LLM API calls wasted
- In my experience, if the agent doesn't construct a meaningful noun phrase to identify the segment. It probably never will.
**With proposed change:**
- Set `max_generations=3`
- Agent stops after 3 unsuccessful attempts
- Runtime reduced. Perhaps enables faster response from a human
This will enable me to quickly assess and analyze the image, without having to wait for the agent grind through many iterations.
---
### Observed Behavior (Example Session)
Below is an abbreviated log from a real session where the agent attempted 12 rounds trying to isolate "wooden deck, not concrete patio":
```
------------------------------ Round 1 ------------------------------
Prompt: "wooden deck" → 1 mask (includes both deck + patio area)
------------------------------ Round 2 ------------------------------
Prompt: "wood flooring" → 1 mask (still includes patio)
------------------------------ Round 3 ------------------------------
Prompt: "outdoor deck" → 1 mask (still includes patio)
...following rounds: similar results with various prompt variations...
```
**Outcome:** The target segment (wooden deck excluding concrete patio) was likely not separable given the image content. Stopping after 3 rounds would have saved 9 unnecessary LLM calls and prevented resource exhaustion.
---
### Benefits
1. **Cost control:** Reduce LLM API costs for batch processing
2. **Faster failures:** Fail fast when target doesn't exist or dificult to identify
3. **Backward compatible:** Default value (100) maintains current behavior
4. **Simple change:** Single parameter addition, minimal code change
Contributor guide
Research direction
Read sam3/agent/inference.py, starting at run_single_image_inference() and its call to agent_inference(). Check how agent_inference() currently receives max_generations, then verify that the new optional argument is forwarded and that the default preserves current behavior. Done means downstream callers can set a lower generation limit without modifying SAM3 source code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100