conductor-oss / conductor-oss/conductor-cli

Stale hardcoded model catalogue: doctor advertises nonexistent Anthropic models; agent init defaults to openai/gpt-4o regardless of configured providers

Open
#103 0 comments 0 reactions 1 assignee Claimed by @ambiorix2099 View on GitHub
agentspan bug
Dominant language
Go
Stars
11
Forks
4
PR merge metrics
No merged PRs in 30d

Description

## Summary

The CLI ships a hardcoded model catalogue that has drifted out of date. Two concrete consequences:

1. `conductor doctor` advertises Anthropic models that **do not exist** — both return 404 from the
Anthropic API.
2. `conductor agent init` always writes `model: openai/gpt-4o` regardless of which providers are
actually configured, so the documented `init` → `run` flow fails out of the box for anyone
without an OpenAI key.

Found while regression-testing CLI branch `cleanup/remove-skill-command` (#100) against server
3.32.0-rc.23.

## 1. `doctor` advertises nonexistent models

`cmd/doctor.go` hardcodes per-provider model lists in `aiProviders`:

```go
{name: "Anthropic", envVars: []string{"ANTHROPIC_API_KEY"},
models: []string{"anthropic/claude-sonnet-4-20250514", "anthropic/claude-3-5-sonnet-20241022"}},
```

`doctor` presents these as available:

```console
$ conductor doctor
AI Providers
ok Anthropic (ANTHROPIC_API_KEY)
anthropic/claude-sonnet-4-20250514
anthropic/claude-3-5-sonnet-20241022
```

Both are gone. Verified against the live Anthropic API with a valid key, with a control:

| Model | `GET /v1/models/` |
|-------|----------------------|
| `claude-sonnet-4-20250514` | **404** |
| `claude-3-5-sonnet-20241022` | **404** |
| `claude-haiku-4-5-20251001` (control) | 200 |

Copying a model straight out of `doctor` output produces a failed agent execution:

```
reasonForIncompletion: Task ... failed: Anthropic Messages API failed with status 404:
{"type":"error","error":{"type":"not_found_error","message":"model: claude-sonnet-4-20250514"}}
```

Note this failure is *server-side at execution time*, so the user gets a FAILED workflow rather than
a validation error — an expensive way to learn the model name was wrong.

The other providers' lists are likely stale too (`openai/gpt-4o-mini`,
`google_gemini/gemini-1.5-pro`, `azure_openai/gpt-4o`); I only verified Anthropic, which is the key
I had.

## 2. `agent init` default ignores configured providers

`cmd/agent.go:38`:

```go
defaultInitModel = "openai/gpt-4o"
```

`agent init` writes that unconditionally:

```console
$ conductor doctor | grep -A1 'AI Providers' -A14 | grep ' ok'
ok Anthropic (ANTHROPIC_API_KEY) # only Anthropic configured

$ conductor agent init probe && cat probe.yaml
model: openai/gpt-4o # unusable here

$ conductor agent run --config probe.yaml "hi"
# fails at execution time — no OPENAI_API_KEY
```

Nothing cross-checks the init default against what `doctor` already knows is configured, even though
both live in the same binary. The command even prints
`Run with: conductor agent run --config probe.yaml "your prompt here"` — an instruction that cannot
succeed.

## Impact

First-run experience for the agent feature is broken in the common case where a user has exactly one
provider key and it isn't OpenAI. The two defects compound: `doctor` says Anthropic is ready and
names two models, both invalid; `agent init` then picks a third model from a provider that isn't
configured at all.

## Fix options

**Model catalogue**

- **A. Drop the model lists from `doctor`.** It only needs to report which providers are *configured*;
naming specific models is a maintenance liability that will drift again. Smallest, most durable
fix.
- **B. Fetch models from the provider at runtime** (e.g. Anthropic's `GET /v1/models`). Accurate, but
adds network calls and per-provider code to a diagnostic command.
- **C. Refresh the hardcoded lists.** Restores correctness today; guarantees a repeat of this issue.

Recommend **A**, or **A** plus a pointer to each provider's model docs.

**`agent init` default**

- **D. Derive the default from configured providers** — reuse the `aiProviders` detection `doctor`
already performs, and pick a model for the provider whose key is present.
- **E. Require `--model`** when the default provider isn't configured, with an error naming what *is*
configured.

Recommend **D** with **E** as the fallback when nothing is configured.

## Test coverage

`cmd/doctor_test.go` exists but does not assert model-string validity (it can't, offline). Worth
noting the model lists are untestable-by-construction while hardcoded, which is an argument for
option A.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.