OpenHands / OpenHands/software-agent-sdk
model-router: drop silent default_model fallback; fail loudly (retryable) on no match
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Context
The classify_and_switch_llm tool routes a task to an LLM profile based on the active meta-profile's classifier. When the classifier fails to produce a usable answer, the tool silently falls back to meta.default_model and switches the conversation to that model — indistinguishable from a successful classification.
This masks a whole class of routing failures as success.
Introduced in #4287 (the Pareto prompt meta-profile routing PR; currently pinned by the saas-deploy preview in OpenHands/saas-deploy#972).
Problem
There are three "didn't get a clean classifier answer" paths in classify_and_switch_llm.py:
- Classifier call throws → returns an error observation (
is_error=True). ✅ Already loud. - Structured mode, classifier returns no in-range integer (
parse_class_index → 0) →target_profile = meta.default_model,chosen_class = None. ❌ Silent. - Direct-prompt mode, reply doesn't match any saved profile (
parse_direct_model → None) →target_profile = meta.default_model,chosen_class = None. ❌ Silent.
Paths 2 and 3 are the issue. A fallback and a real match produce identical observations, so:
- You cannot tell from the transcript that routing gave up.
- A single malformed JSON reply from the Pareto prompt classifier quietly drops the conversation onto
minimax-m3(the currentdefault_model) and leaves it there — a silent regression with no signal to retry or escalate.
Proposal
Remove default_model from MetaProfile and make the no-match paths fail loudly and retryably:
- On no-match / unparseable reply, return a
ClassifyAndSwitchLLMObservation.from_text(..., is_error=True)— reusing the existing error-observation pattern the file already uses for classifier exceptions (the "Classifier call failed" path). - This makes the failure visible in the transcript and retryable at the agent/loop level, rather than masked as a successful route to the default.
A loud failure is strictly better than a silent fallback: the silent fallback also aborts real routing, it just does so invisibly and then keeps going on the wrong model. A loud failure surfaces it for retry or user escalation.
Open question / decision needed
Should "no match" be a hard error (abort the tool call, retryable) or an explicitly-tagged fallback observation (keep a default but mark chosen_class = "fallback: no match" / an is_fallback flag)? This issue proposes the hard-fail approach. If resilience to a single flaky classifier reply is a concern, the tagged-fallback is the smaller alternative — but the current silent behavior is strictly worse than either.
Scope
SDK (software-agent-sdk, this repo)
openhands/sdk/llm/meta_profile_store.py— removedefault_modelfield (L70–72) + summary dict entry (L274); update the docstring invariant (L4).openhands/sdk/tool/builtins/classify_and_switch_llm.py— replace the twotarget_profile = meta.default_modelassignments (L333, L348) withis_error=Trueobservations; update docstring (L7, L112).openhands/agent_server/meta_profiles_router.py— removedefault_modelfrom the API schema (L53).- Tests — update
test_meta_profile_store.py,test_classify_and_switch_llm.py,test_meta_profiles_router.pyto assert error-on-no-match instead of fallback.
Frontend follow-up (OpenHands/OpenHands, #16147)
src/api/meta-profiles-service/meta-profiles-service.api.ts— dropdefault_modelfromMetaProfile/MetaProfileInfo(L42, L54) + docstring.src/components/features/settings/meta-llm-profiles/meta-profile-editor.tsx— remove the "Default model" dropdown UI (~L246–268), state/load/save wiring (L57, L73, L166, L184).src/components/features/settings/meta-llm-profiles/default-meta-profile.ts— dropdefault_modelfrom the two built-in profiles (L80, L138).src/components/features/settings/meta-llm-profiles/{router-profiles.ts,meta-profile-row.tsx,meta-llm-settings-view.tsx}— drop field/type/usages.src/i18n/translation.json— removeSETTINGS$META_PROFILE_DEFAULT+SETTINGS$META_PROFILE_DEFAULT_HELP(~33 lines).- Tests across
__tests__/.../meta-llm-profiles/*.
Out of scope
The unrelated ACP per-provider default_model (openhands/sdk/settings/acp_providers.py, src/constants/acp-providers.ts, option-service, llm-settings.tsx) is a different concept and is not touched.
Line impact: ~7 production + ~22 test lines in the SDK; ~66 production + ~9 test lines in the frontend follow-up. The two target_profile = meta.default_model lines become error returns rather than pure deletions.
This issue was created by an AI agent (OpenHands) on behalf of Juan Michelini.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with openhands/sdk/tool/builtins/classify_and_switch_llm.py and its no-match paths, then read openhands/sdk/llm/meta_profile_store.py and openhands/agent_server/meta_profiles_router.py. Run the relevant tests in test_classify_and_switch_llm.py, test_meta_profile_store.py, and test_meta_profiles_router.py. Done means no-match responses are error observations, default_model is removed from the SDK schema and store, and the tests assert the retryable failure behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100