conductor-oss / conductor-oss/conductor-cli
Stale hardcoded model catalogue: doctor advertises nonexistent Anthropic models; agent init defaults to openai/gpt-4o regardless of configured providers
- Dominant language
- Go
- Stars
- 11
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
The CLI ships a hardcoded model catalogue that has drifted out of date. Two concrete consequences:
1. `conductor doctor` advertises Anthropic models that **do not exist** — both return 404 from the
Anthropic API.
2. `conductor agent init` always writes `model: openai/gpt-4o` regardless of which providers are
actually configured, so the documented `init` → `run` flow fails out of the box for anyone
without an OpenAI key.
Found while regression-testing CLI branch `cleanup/remove-skill-command` (#100) against server
3.32.0-rc.23.
## 1. `doctor` advertises nonexistent models
`cmd/doctor.go` hardcodes per-provider model lists in `aiProviders`:
```go
{name: "Anthropic", envVars: []string{"ANTHROPIC_API_KEY"},
models: []string{"anthropic/claude-sonnet-4-20250514", "anthropic/claude-3-5-sonnet-20241022"}},
```
`doctor` presents these as available:
```console
$ conductor doctor
AI Providers
ok Anthropic (ANTHROPIC_API_KEY)
anthropic/claude-sonnet-4-20250514
anthropic/claude-3-5-sonnet-20241022
```
Both are gone. Verified against the live Anthropic API with a valid key, with a control:
| Model | `GET /v1/models/` |
|-------|----------------------|
| `claude-sonnet-4-20250514` | **404** |
| `claude-3-5-sonnet-20241022` | **404** |
| `claude-haiku-4-5-20251001` (control) | 200 |
Copying a model straight out of `doctor` output produces a failed agent execution:
```
reasonForIncompletion: Task ... failed: Anthropic Messages API failed with status 404:
{"type":"error","error":{"type":"not_found_error","message":"model: claude-sonnet-4-20250514"}}
```
Note this failure is *server-side at execution time*, so the user gets a FAILED workflow rather than
a validation error — an expensive way to learn the model name was wrong.
The other providers' lists are likely stale too (`openai/gpt-4o-mini`,
`google_gemini/gemini-1.5-pro`, `azure_openai/gpt-4o`); I only verified Anthropic, which is the key
I had.
## 2. `agent init` default ignores configured providers
`cmd/agent.go:38`:
```go
defaultInitModel = "openai/gpt-4o"
```
`agent init` writes that unconditionally:
```console
$ conductor doctor | grep -A1 'AI Providers' -A14 | grep ' ok'
ok Anthropic (ANTHROPIC_API_KEY) # only Anthropic configured
$ conductor agent init probe && cat probe.yaml
model: openai/gpt-4o # unusable here
$ conductor agent run --config probe.yaml "hi"
# fails at execution time — no OPENAI_API_KEY
```
Nothing cross-checks the init default against what `doctor` already knows is configured, even though
both live in the same binary. The command even prints
`Run with: conductor agent run --config probe.yaml "your prompt here"` — an instruction that cannot
succeed.
## Impact
First-run experience for the agent feature is broken in the common case where a user has exactly one
provider key and it isn't OpenAI. The two defects compound: `doctor` says Anthropic is ready and
names two models, both invalid; `agent init` then picks a third model from a provider that isn't
configured at all.
## Fix options
**Model catalogue**
- **A. Drop the model lists from `doctor`.** It only needs to report which providers are *configured*;
naming specific models is a maintenance liability that will drift again. Smallest, most durable
fix.
- **B. Fetch models from the provider at runtime** (e.g. Anthropic's `GET /v1/models`). Accurate, but
adds network calls and per-provider code to a diagnostic command.
- **C. Refresh the hardcoded lists.** Restores correctness today; guarantees a repeat of this issue.
Recommend **A**, or **A** plus a pointer to each provider's model docs.
**`agent init` default**
- **D. Derive the default from configured providers** — reuse the `aiProviders` detection `doctor`
already performs, and pick a model for the provider whose key is present.
- **E. Require `--model`** when the default provider isn't configured, with an error naming what *is*
configured.
Recommend **D** with **E** as the fallback when nothing is configured.
## Test coverage
`cmd/doctor_test.go` exists but does not assert model-string validity (it can't, offline). Worth
noting the model lists are untestable-by-construction while hardcoded, which is an argument for
option A.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.