galaxyproject / galaxyproject/loom
Model switch reports success but selected Claude model 404s on first call
- Dominant language
- TypeScript
- Stars
- 14
- Forks
- 12
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 17
Description
## What happens
Several Claude models offered in Orbit's model picker can be "switched to" successfully -- Orbit shows "Changed model... Agent restarted" and marks the model active -- but the **first real API call returns `404 not_found`**. The switch is only validated lazily, so a broken selection looks healthy until a turn fails mid-work.
## Repro / context
Tested with the user's own Anthropic API key (Orbit 0.3.1):
| Model selected | Result |
|---------------------------|-------------------|
| `claude-3-5-haiku-latest` | 404 `not_found` |
| `claude-3-5-haiku-20241022` | 404 `not_found` |
| `claude-sonnet-4-6` | works |
| `claude-haiku-4-5-20251001` | works |
`/cost` correctly shows 0 tokens / $0 for the two failing models, confirming the calls never executed. The activity log shows the failing model IDs as bare `agent turn start` entries with no tool calls, while working models proceed normally.
## Expected vs actual
- Expected: switching to a model the account/key can't call should fail the switch loudly (or the model shouldn't be offered) -- surfaced at switch time, not as a mid-turn 404.
- Actual: switch reports success and shows the model as active; the 404 only appears when the next inference runs.
## Notes
- Likely related to the picker being populated from the bundled registry rather than what's actually callable for this credential. See #236 (derive picker from the provider's live `/models` endpoint) for the root-cause fix and #235 (surface silent provider fallback). This issue is specifically about **validating the selection before declaring the switch successful** -- here there is no fallback; the broken model stays active and 404s.
- `claude-3-5-haiku-*` are real published models, so this is account/key/region access (or alias-mapping), not simply stale IDs. The fix is the same either way: don't present an unusable model as switched-and-active.
## Sysinfo
Orbit 0.3.1 - darwin/arm64 - electron 41.2.1 - provider anthropic - models tested: claude-3-5-haiku-latest, claude-3-5-haiku-20241022 (fail); claude-sonnet-4-6, claude-haiku-4-5-20251001 (work)
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace the model picker and switch flow, then follow the Anthropic provider request for the listed Claude model IDs. Reproduce with a failing model and a working model from the issue. Done means an unavailable selection is rejected or surfaced before it is marked active, rather than failing on the first inference call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- electron, typescript
- Domain
- api, frontend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100