step-3.7-flash: vision support not detected at runtime (LiteLLM proxy /v1/model/info lookup returns None)

Summary

For step-3.7-flash (and likely any model whose supports_vision flag is set only via the eval LiteLLM proxy's model_info, not via upstream LiteLLM static metadata), the integration test t08_image_file_viewing is silently skipped because the SDK's LLM.vision_is_active() returns False at runtime.

Root cause has been narrowed in PR #3477: LLM._model_info is None because _get_model_info_from_litellm_proxy() silently fails its lookup against the eval proxy.

This issue is a non-blocker for #3476 / PR #3477 — text and tool-use paths for step-3.7-flash work at 100% in integration. It only affects whether the SDK believes the model is vision-capable enough to actually execute the vision test.

What we know

The eval proxy is configured with:

- model_name: "step-3.7-flash"
  litellm_params:
    model: "openai/step-3.7-flash"
    api_base: https://api.stepfun.ai/v1
  model_info:
    supports_vision: true

A temporary diagnostic (reverted in PR #3477) printed for every test fixture:

[diagnostic PR #3477] model=litellm_proxy/step-3.7-flash model_info=None vision_is_active=False

So even with LLM.model == "litellm_proxy/step-3.7-flash" (lookup key step-3.7-flash matching the proxy's model_name), _get_model_info_from_litellm_proxy returns None.

The SDK gate that decides vision support is:

def _supports_vision(self) -> bool:
    return (
        supports_vision(model_for_caps)
        or supports_vision(model_for_caps.split("/")[-1])
        or (self._model_info is not None
            and self._model_info.get("supports_vision", False))
        or False
    )

All three OR branches resolve to False:

1. supports_vision("openrouter/stepfun/step-3.7-flash") → False (LiteLLM upstream metadata gap).
2. supports_vision("step-3.7-flash") → False (same gap).
3. _model_info is None → the proxy fallback short-circuits.

So the test correctly (per the SDK's view of reality) raises SkipTest.

Most likely root causes (in order)

1. /v1/model/info requires master/admin key, not the chat-completions virtual key. LiteLLM proxy commonly gates the model-info endpoint to admin tokens only. If LLM_API_KEY_EVAL is a virtual key, the GET returns 401 and the SDK silently falls through (_get_model_info_from_litellm_proxy wraps everything in try/except and only logs at debug level).
2. The endpoint returns a different JSON shape than {"data": [{"model_name": ..., "model_info": {...}}]}.
3. The entry isn't actually deployed under that model_name in the live proxy config.

Suggested next steps

Pick whichever is cheapest:

• Probe /v1/model/info with the eval runner's LLM_API_KEY from an admin context: log status_code, body length, and whether "step-3.7-flash" appears in the response. PR #3477 had a probe pushed in 566d9ab9 that can be cherry-picked or re-implemented here.
• If it's an auth issue: grant the eval virtual key access to /v1/model/info, or raise the SDK's silent log level so this kind of plumbing failure surfaces by default.
• Consider extending LLM with an explicit supports_vision opt-in (parallel to disable_vision) so model registrations can assert capabilities locally without depending on either upstream LiteLLM metadata or a runtime proxy lookup.

Acceptance

Either:

• litellm_proxy/step-3.7-flash runs t08_image_file_viewing (no skip) and passes, or
• the SDK exposes a way to declare supports_vision=true at registration time so the integration test executes.

Context

• Original issue: #3476
• Original PR: #3477
• Provider docs: native multimodal (196B LLM + 1.8B ViT) — https://static.stepfun.com/blog/step-3.7-flash/

---

This issue was created by an AI agent (OpenHands) on behalf of @juanmichelini.

Start with LLM._get_model_info_from_litellm_proxy and the probe from PR #3477 commit 566d9ab9; inspect the eval proxy's /v1/model/info response using the eval API key. Then run integration test t08_image_file_viewing for litellm_proxy/step-3.7-flash. Done means the test runs and passes, or the SDK provides an explicit supports_vision registration option.