CommunityToolkit / CommunityToolkit/Aspire
vLLM hosting and client integration
- Dominant language
- C#
- Stars
- 627
- Forks
- 196
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 35
Description
### Aspire issue link
_No response_ (not previously discussed in dotnet/aspire)
### Overview
vLLM is a popular, OpenAI-compatible inference server for running open-weight models on your own
GPU. An Aspire app host has no first-class way to run one today. You drop down to a raw container
resource and hand-write the image, the port, the GPU runtime args, a readiness health check, and
then the OpenAI client wiring on the consuming side. Every app repeats that setup, and the first run
has a few sharp edges: the health check has to gate on the model actually loading, and the OpenAI
client needs the `/v1` path plus a non-empty API key even though vLLM ignores the key.
The Community Toolkit already covers this shape for Ollama. vLLM is the natural companion for
GPU-served models, higher throughput, and larger context windows, so I'd like to add it as a
matching pair of packages:
- **`CommunityToolkit.Aspire.Hosting.VLLM`** (hosting). `AddVLLM("vllm")` adds the
`vllm/vllm-openai` container with an HTTP endpoint (container port 8000) and a `/health` check
that reports healthy only once the model is loaded, so `WaitFor` dependents block until the server
can actually answer. `WithGPUSupport()` adds `--gpus all` for NVIDIA or switches to the ROCm image
with the right device mounts for AMD. `WithDataVolume()` persists the Hugging Face cache across
restarts, `WithModel(...)` and `WithServedModelName(...)` choose and name the model, and
`WithHuggingFaceToken(...)` wires a secret parameter for gated models.
- **`CommunityToolkit.Aspire.VLLM`** (client). `AddVLLMClient("vllm").AddChatClient()` reads the
hosting resource's connection string and registers an `IChatClient` (through
`Microsoft.Extensions.AI`) pointed at the server. It appends `/v1`, supplies the placeholder API
key the OpenAI client requires, targets the served model name, and registers a matching `/health`
check plus OpenTelemetry. Because the surface is `Microsoft.Extensions.AI`, the same app code keeps
working against any OpenAI-compatible endpoint later.
The two mirror `CommunityToolkit.Aspire.Hosting.Ollama` and `CommunityToolkit.Aspire.OllamaSharp`:
add the server in the app host, reference it from a project, and consume it as an `IChatClient`.
### Usage example
Add the server in the app host and gate a project on it:
```csharp
// AppHost.cs
var builder = DistributedApplication.CreateBuilder(args);
var vllm = builder.AddVLLM("vllm")
.WithGPUSupport() // NVIDIA by default; VLLMGpuVendor.AMD for ROCm
.WithDataVolume() // persist the Hugging Face cache
.WithModel("Qwen/Qwen3-8B");
builder.AddProject("web")
.WithReference(vllm) // injects the connection string
.WaitFor(vllm); // blocks until the model is serving
builder.Build().Run();
```
Consume it as an `IChatClient` in the referencing project:
```csharp
// Web Program.cs
builder.AddVLLMClient("vllm", settings => settings.Model = "Qwen/Qwen3-8B")
.AddChatClient()
.UseFunctionInvocation();
```
### Additional context
Both packages already exist, build, and run. I built them for
[`magenticbrain-vllm-aspire`](https://github.com/luisquintanilla/magenticbrain-vllm-aspire), a local
RAG app, and shaped them from the start to match this repo's conventions so they can move here with
little change.
**Proven running.** This is not a sketch. The same two packages run
[`microsoft/MagenticBrain`](https://huggingface.co/microsoft/MagenticBrain), a 14B model quantized to
4-bit, on a single 16 GB laptop GPU (RTX 3080), under Aspire, behind the `aichatweb` RAG template,
with tool-calling. I verified the full client path against a live GPU server: the placeholder key,
the `/v1` suffix, and the served model name all work, and the app returns grounded answers with
citations.
**What's already there** (links to `main`):
- Hosting integration:
[`src/CommunityToolkit.Aspire.Hosting.VLLM`](https://github.com/luisquintanilla/magenticbrain-vllm-aspire/tree/main/src/CommunityToolkit.Aspire.Hosting.VLLM)
(namespace `Aspire.Hosting`, the `AddVLLM` surface, `VLLMResource : ContainerResource,
IResourceWithConnectionString, IResourceWithEndpoints`).
- Client integration:
[`src/CommunityToolkit.Aspire.VLLM`](https://github.com/luisquintanilla/magenticbrain-vllm-aspire/tree/main/src/CommunityToolkit.Aspire.VLLM)
(extensions in `Microsoft.Extensions.Hosting`, per the client-integration convention).
- Tests: 9 hosting unit tests and 15 client unit tests, plus a Docker-gated app host test.
- Examples: an example app host and an example consumer under
[`examples/vllm`](https://github.com/luisquintanilla/magenticbrain-vllm-aspire/tree/main/examples/vllm).
- A porting checklist that maps every file to this repo's layout and reconciles it against
[`docs/create-integration.md`](https://github.com/CommunityToolkit/Aspire/blob/main/docs/create-integration.md):
[`docs/upstream-vllm-integration.md`](https://github.com/luisquintanilla/magenticbrain-vllm-aspire/blob/main/docs/upstream-vllm-integration.md).
**Why a dedicated client and not just `AddOpenAIClient`?** vLLM is OpenAI-compatible, so a reviewer
may reasonably ask. The client adds three vLLM-specific things on top of the generic OpenAI client:
it pairs with the hosting resource over the connection string and appends `/v1` (the hosting
resource emits the base URL without it, so a raw OpenAI client 404s until you fix it up by hand); it
fills in vLLM-correct defaults (the placeholder API key and the served model name); and it adds
vLLM's `/health` readiness check. If you'd prefer, the same behavior could ship as a defaults layer
over `AddOpenAIClient` rather than a standalone package. Happy to go either way.
**Alternatives.** Today you write `AddContainer("vllm/vllm-openai", ...)` by hand and pair it with
`AddOpenAIClient`, reproducing the GPU args, the readiness health check, and the `/v1` and API-key
fix-ups in every app. That works, but it is the boilerplate an integration should own.
**Relationship to Ollama.** This complements the existing Ollama integration rather than replacing
it. Ollama is the easy CPU/GGUF path; vLLM is the GPU-first, high-throughput, OpenAI-native path for
larger models and longer context. Many apps will want vLLM for chat and Ollama for embeddings.
**A few choices for maintainers.** I'm flexible on all of these:
- Acronym casing: `VLLM` (matches the product) vs `Vllm` (matches .NET acronym casing), for both
type names and `AddVLLM`/`AddVllm`.
- The `VLLMGpuVendor` enum name (Ollama uses an unprefixed `GpuVendor`; I prefixed it to avoid a
clash if both integrations are referenced from the same app host).
- Client shape: `AddVLLMClient(name).AddChatClient()` (leaves room for `.AddEmbeddingGenerator()`,
since vLLM can serve embedding models too) vs a single-call `AddVLLMChatClient(name)`.
- One combined proposal vs separate hosting and client issues, whichever you track better.
I'll sign the .NET Foundation CLA and can open the companion docs PR to `aspire.dev`. I'd love to
bring this in.
### Help us help you
Yes, I'd like to be assigned to work on this item
Contributor guide
Research direction
Start with docs/create-integration.md and the porting checklist in docs/upstream-vllm-integration.md. Compare the referenced CommunityToolkit.Aspire.Hosting.VLLM and CommunityToolkit.Aspire.VLLM source, their 9 hosting and 15 client tests, and the examples/vllm app; done means the packages, tests, examples, and companion documentation conform to this repository's conventions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, docker
- Domain
- ai, backend, devtools
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100