aaif-goose / aaif-goose/goose

Support local model runners that distribute models as OCI artifacts

Open
#11,692 10 comments 0 reactions 1 assignee Claimed by @ericcurtin View on GitHub
Dominant language
Rust
Stars
54.2k
Forks
6.2k
Avg merge
3d 4h
Merged PRs (30d)
240

Description

**What problem would this solve?**

Goose's built-in local options today are Ollama and, via the OpenAI-compatible path, servers like vLLM and LM Studio. A separate class of local runner has appeared that distributes model weights as **OCI artifacts** pulled from a normal container registry, rather than through a bespoke model library. [llmman](https://github.com/llmmanorg/llmman) is one; the packaging format is the [CNCF ModelPack spec](https://github.com/modelpack/model-spec).

Who this affects: people in environments where the container registry is already the governed distribution channel — mirrored, scanned, access-controlled, air-gapped. For them, "pull the model the same way you pull an image" is the difference between a model being usable and not. Right now they have to configure Goose's generic OpenAI-compatible provider by hand, which works but means the runner isn't discoverable in `goose configure`, and its default port and no-API-key behaviour aren't known to Goose.

The narrower version of the problem: Goose has no first-class local provider whose model identifiers are registry references.

**What would a good outcome look like?**

Someone running such a server locally can select it during `goose configure` and start a session without hand-editing config or knowing which generic provider to bend into shape. Its defaults — port, no API key required — are known to Goose, and model identifiers are whatever the runner resolves against its registry rather than a list Goose maintains.

Beyond that I'd rather not prescribe the shape. There's a real design question about whether this deserves its own provider or is better served by improving the generic OpenAI-compatible path, and I don't have a strong view on which is right for the project.

**Possible approaches**

Constraints and trade-offs worth weighing in the design discussion:

- These runners serve an OpenAI-compatible API, so no new wire format is needed. That makes the marginal implementation cost low — but also raises the fair question of whether a distinct provider earns its long-term maintenance cost versus documenting the generic path better.
- Model lists can't be static. Available models are whatever the user has pulled, so discovery has to come from the server's `/v1/models` or be left open.
- Some of these servers also speak the Anthropic Messages API, which may or may not be interesting to Goose.
- There is more than one such runner, so if this is worth doing it's probably worth designing for the category rather than one product.

**Additional context**

I have a working implementation on a local branch (a provider definition in the existing pattern; `cargo check` and `cargo fmt` clean) which I'm deliberately **not** opening as a PR, since CONTRIBUTING is clear that implementation waits for **Ready**. Happy to attach it as a reference diff in this issue if that would help the design discussion, or to throw it away if the conclusion is that the generic path is the better answer.

Disclosure: I used an AI assistant to research the codebase and draft this. I've reviewed it and can speak to any of it.

- [x] I have verified this does not duplicate an existing feature request

Contributor guide

Open the contributing guide

Research direction

Look at the existing provider definitions in the codebase, likely under a `providers` or `local` module. Examine how Ollama and the generic OpenAI-compatible provider are implemented. The design discussion centers on whether to create a new provider or extend the generic one. Start by understanding the configuration flow in `goose configure` and how model discovery works. 'Done' means a user can select this runner type during configuration without manual edits.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, rust
Domain
ai-infra-agents, backend-api-design, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.