facebookresearch / facebookresearch/sam3

Add `max_generations` parameter to `run_single_image_inference()` to allow configurable agent iteration limit

Open
#446 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.7k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

### Summary

The SAM3 agent's `agent_inference()` function supports a `max_generations` parameter (default: 100) to limit the number of agent rounds. However, `run_single_image_inference()` does not expose this parameter, making it impossible for downstream users to configure a lower limit without modifying SAM3 source code.

This enhancement would enable users to set sensible early-stopping limits for use cases where the target segment may not exist in the image, preventing excessive LLM API calls and runtime.

---

### Problem Statement

When using the SAM3 agent via `run_single_image_inference()`, there is no way to control the maximum number of segmentation rounds the agent will attempt. The agent will continue iterating up to the hardcoded default of 100 generations (defined in `agent_inference()`).

**Use case:** A segmentation pipeline where the target object may not always be present or semi covered, the agent can waste a lot of time and API costs trying multiple prompt variations when a target simply doesn't exist. Therefore, I request a way to configure a lower threshold (e.g., 3-5 rounds) to fail fast.

---

### Proposed Solution

Add an optional `max_generations` parameter to `run_single_image_inference()` and pass it through to `agent_inference()`.

**Files to modify:**
- `sam3/agent/inference.py`

**Proposed changes:**

```python
def run_single_image_inference(
image_path,
text_prompt,
llm_config,
send_generate_request,
call_sam_service,
output_dir="agent_output",
debug=False,
max_generations=100, # NEW: Add parameter with default matching agent_inference
):
"""Run inference on a single image with provided prompt

Args:
...existing args...
max_generations: Maximum number of agent rounds to attempt (default: 100)
"""

# ...existing setup code...

agent_history, final_output_dict, rendered_final_output = agent_inference(
image_path,
text_prompt,
send_generate_request=send_generate_request,
call_sam_service=call_sam_service,
output_dir=output_dir,
debug=debug,
max_generations=max_generations, # NEW: Pass through parameter
)

# ...existing save code...
```

---

### Example Usage (After Change)

**Current (no control):**
```python
from sam3.agent.inference import run_single_image_inference

# Agent will try up to 100 rounds (hardcoded default)
output_image_path, output_json_path = run_single_image_inference(
image_path="house_exterior.jpg",
text_prompt="wooden deck, not concrete patio",
llm_config=config,
send_generate_request=my_llm_func,
call_sam_service=my_sam_func,
)
```

**Proposed (configurable):**
```python
from sam3.agent.inference import run_single_image_inference

# Agent will stop after 3 unsuccessful rounds
output_image_path, output_json_path = run_single_image_inference(
image_path="house_exterior.jpg",
text_prompt="wooden deck, not concrete patio",
llm_config=config,
send_generate_request=my_llm_func,
call_sam_service=my_sam_func,
max_generations=3, # NEW: User can specify limit
)
```

---

### Real-World Example

**Scenario:** Processing a batch of 1,000 real estate images for property analysis. Some images contain the target segment (e.g., "wooden deck"), some don't (e.g., houses without decks, obstructed views) and the rest has a mix wooden deck and other.

**Current behavior:**
- Agent tries up to 100 rounds per image
- For complicated images, e.g., without the target, this potentially returns ~100 LLM API calls wasted
- In my experience, if the agent doesn't construct a meaningful noun phrase to identify the segment. It probably never will.

**With proposed change:**
- Set `max_generations=3`
- Agent stops after 3 unsuccessful attempts
- Runtime reduced. Perhaps enables faster response from a human

This will enable me to quickly assess and analyze the image, without having to wait for the agent grind through many iterations.

---

### Observed Behavior (Example Session)

Below is an abbreviated log from a real session where the agent attempted 12 rounds trying to isolate "wooden deck, not concrete patio":

```
------------------------------ Round 1 ------------------------------
Prompt: "wooden deck" → 1 mask (includes both deck + patio area)

------------------------------ Round 2 ------------------------------
Prompt: "wood flooring" → 1 mask (still includes patio)

------------------------------ Round 3 ------------------------------
Prompt: "outdoor deck" → 1 mask (still includes patio)

...following rounds: similar results with various prompt variations...

```

**Outcome:** The target segment (wooden deck excluding concrete patio) was likely not separable given the image content. Stopping after 3 rounds would have saved 9 unnecessary LLM calls and prevented resource exhaustion.

---

### Benefits

1. **Cost control:** Reduce LLM API costs for batch processing
2. **Faster failures:** Fail fast when target doesn't exist or dificult to identify
3. **Backward compatible:** Default value (100) maintains current behavior
4. **Simple change:** Single parameter addition, minimal code change

Contributor guide

Open the contributing guide

Research direction

Read sam3/agent/inference.py, starting at run_single_image_inference() and its call to agent_inference(). Check how agent_inference() currently receives max_generations, then verify that the new optional argument is forwarded and that the default preserves current behavior. Done means downstream callers can set a lower generation limit without modifying SAM3 source code.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.