OpenHands / OpenHands/software-agent-sdk

model-router: drop silent default_model fallback; fail loudly (retryable) on no match

Open
#5,068 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement llm sdk tools
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Context

The classify_and_switch_llm tool routes a task to an LLM profile based on the active meta-profile's classifier. When the classifier fails to produce a usable answer, the tool silently falls back to meta.default_model and switches the conversation to that model — indistinguishable from a successful classification.

This masks a whole class of routing failures as success.

Introduced in #4287 (the Pareto prompt meta-profile routing PR; currently pinned by the saas-deploy preview in OpenHands/saas-deploy#972).

Problem

There are three "didn't get a clean classifier answer" paths in classify_and_switch_llm.py:

  1. Classifier call throws → returns an error observation (is_error=True). ✅ Already loud.
  2. Structured mode, classifier returns no in-range integer (parse_class_index → 0) → target_profile = meta.default_model, chosen_class = None. ❌ Silent.
  3. Direct-prompt mode, reply doesn't match any saved profile (parse_direct_model → None) → target_profile = meta.default_model, chosen_class = None. ❌ Silent.

Paths 2 and 3 are the issue. A fallback and a real match produce identical observations, so:

  • You cannot tell from the transcript that routing gave up.
  • A single malformed JSON reply from the Pareto prompt classifier quietly drops the conversation onto minimax-m3 (the current default_model) and leaves it there — a silent regression with no signal to retry or escalate.

Proposal

Remove default_model from MetaProfile and make the no-match paths fail loudly and retryably:

  • On no-match / unparseable reply, return a ClassifyAndSwitchLLMObservation.from_text(..., is_error=True) — reusing the existing error-observation pattern the file already uses for classifier exceptions (the "Classifier call failed" path).
  • This makes the failure visible in the transcript and retryable at the agent/loop level, rather than masked as a successful route to the default.

A loud failure is strictly better than a silent fallback: the silent fallback also aborts real routing, it just does so invisibly and then keeps going on the wrong model. A loud failure surfaces it for retry or user escalation.

Open question / decision needed

Should "no match" be a hard error (abort the tool call, retryable) or an explicitly-tagged fallback observation (keep a default but mark chosen_class = "fallback: no match" / an is_fallback flag)? This issue proposes the hard-fail approach. If resilience to a single flaky classifier reply is a concern, the tagged-fallback is the smaller alternative — but the current silent behavior is strictly worse than either.

Scope

SDK (software-agent-sdk, this repo)
  • openhands/sdk/llm/meta_profile_store.py — remove default_model field (L70–72) + summary dict entry (L274); update the docstring invariant (L4).
  • openhands/sdk/tool/builtins/classify_and_switch_llm.py — replace the two target_profile = meta.default_model assignments (L333, L348) with is_error=True observations; update docstring (L7, L112).
  • openhands/agent_server/meta_profiles_router.py — remove default_model from the API schema (L53).
  • Tests — update test_meta_profile_store.py, test_classify_and_switch_llm.py, test_meta_profiles_router.py to assert error-on-no-match instead of fallback.
Frontend follow-up (OpenHands/OpenHands, #16147)
  • src/api/meta-profiles-service/meta-profiles-service.api.ts — drop default_model from MetaProfile / MetaProfileInfo (L42, L54) + docstring.
  • src/components/features/settings/meta-llm-profiles/meta-profile-editor.tsx — remove the "Default model" dropdown UI (~L246–268), state/load/save wiring (L57, L73, L166, L184).
  • src/components/features/settings/meta-llm-profiles/default-meta-profile.ts — drop default_model from the two built-in profiles (L80, L138).
  • src/components/features/settings/meta-llm-profiles/{router-profiles.ts,meta-profile-row.tsx,meta-llm-settings-view.tsx} — drop field/type/usages.
  • src/i18n/translation.json — remove SETTINGS$META_PROFILE_DEFAULT + SETTINGS$META_PROFILE_DEFAULT_HELP (~33 lines).
  • Tests across __tests__/.../meta-llm-profiles/*.

Out of scope

The unrelated ACP per-provider default_model (openhands/sdk/settings/acp_providers.py, src/constants/acp-providers.ts, option-service, llm-settings.tsx) is a different concept and is not touched.


Line impact: ~7 production + ~22 test lines in the SDK; ~66 production + ~9 test lines in the frontend follow-up. The two target_profile = meta.default_model lines become error returns rather than pure deletions.

This issue was created by an AI agent (OpenHands) on behalf of Juan Michelini.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with openhands/sdk/tool/builtins/classify_and_switch_llm.py and its no-match paths, then read openhands/sdk/llm/meta_profile_store.py and openhands/agent_server/meta_profiles_router.py. Run the relevant tests in test_classify_and_switch_llm.py, test_meta_profile_store.py, and test_meta_profiles_router.py. Done means no-match responses are error observations, default_model is removed from the SDK schema and store, and the tests assert the retryable failure behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.