segment_sam3_text_prompt returns [] on transport failure, making a SAM3 timeout indistinguishable from no detections

Summary

segment_sam3_text_prompt converts a transport failure into an empty result. A SAM3 timeout, a dead server, or any other exception from post_with_retries returns [], which is indistinguishable from "the model matched nothing". This function is part of the API surface injected into generated code, so during a scored run an infrastructure fault is recorded as a perception result.

Where

cap/integrations/vision/sam3.py:70-79

try:
    resp = post_with_retries(f"{SERVICE_URL}/segment", payload)
    results_data = resp["results"]
except Exception as e:
    print(f"Failed to communicate with SAM3 service at {SERVICE_URL}: {e}")
    return []

The sibling function in the same file takes the opposite approach — segment_sam3_point_prompt at line 208-211 catches RuntimeError and re-raises it. The two paths have inconsistent failure semantics.

segment_sam3_text_prompt is registered into the injected function table at cap/integrations/franka/libero_reduced.py:102 and cap/integrations/franka/control_reduced.py:103.

How I hit it

Running the optional post-server smoke test from the README, on a first run right after the perception servers came up:

Failed to communicate with SAM3 service at http://127.0.0.1:8114:
  Request to http://127.0.0.1:8114/segment failed after 1 retries / 120.00s.
  Last error: HTTPConnectionPool(host='127.0.0.1', port=8114): Read timed out. (read timeout=120.0)
[segment_sam3_text_prompt] SAM3 returned no results for prompt: 'bowl'
>>> SMOKE_RESULT (512, 800, 3) 0

Re-running the identical command a few minutes later, with nothing changed but a warm model:

>>> SMOKE_RESULT (512, 800, 3) 200

Same frame, same prompt: 0 masks versus 200. The only difference was a swallowed timeout.

Note that post_with_retries treats timeout_seconds=120.0 as a total wall-clock budget, so a single 120 s read timeout exhausts it and there is effectively one attempt — hence "failed after 1 retries".

Why it is worth fixing rather than tuning

The printed message does reach the fix-loop, since generated-code stdout is fed back to the model, so this is not completely invisible. But the trial still ends as a task failure, and nothing in the recorded result distinguishes "SAM3 was slow" from "the object was not visible". For a benchmark whose headline number is task success rate, a perception outage during a long run biases results downward in a way that no downstream analysis of the traces can detect after the fact.

Suggested fix

Raise on transport failure, matching segment_sam3_point_prompt, so a trial fails loudly instead of scoring as a negative detection. If swallowing is deliberate for robustness, returning a distinguishable sentinel — or recording the fault in the trial trace — would at least let analysis separate the two cases.

Separately, a cold first request is easy to hit: the model finishes loading lazily, and the very first /segment can exceed 120 s on a contended GPU. A warm-up request at server startup, or a longer budget for the first call, would remove the common case.

Environment

Fresh clone at f4c8939, Linux x86-64, .venv-libero (Python 3.12). SAM3 on a shared H100 alongside two other perception models; facebook/sam3 weights freshly downloaded.

Start in cap/integrations/vision/sam3.py:70-79 and compare segment_sam3_text_prompt with segment_sam3_point_prompt at lines 208-211. Check the injected registrations in cap/integrations/franka/libero_reduced.py:102 and control_reduced.py:103, then reproduce the optional post-server smoke test from the README. Done means transport failures are distinguishable from genuine empty detections, with the existing API paths behaving consistently.