[dotnet-ai] ONNX Runtime + local LLM inference
- Dominant language
- C#
- Stars
- 5.4k
- Forks
- 415
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 84
Description
## Parent Issue
Part of the `dotnet-ai` plugin proposal: #225
## Summary
Add the `onnx-runtime-inference` and `local-llm-inference` skills to the `dotnet-ai` plugin, including one reference file.
These two skills cover model execution — running pre-trained models and LLMs locally. They are submitted together because they share the deployment-at-the-edge theme and reference each other in their routing guidance.
## Skills in This Issue
### onnx-runtime-inference
Guides developers through running pre-trained ONNX models for inference in .NET, both standalone (via `Microsoft.ML.OnnxRuntime`) and through ML.NET (via `Microsoft.ML.OnnxTransformer`).
**Covers:**
- **Standalone approach** — `InferenceSession`, input/output tensor construction, execution
- **ML.NET approach** — `ApplyOnnxModel()` within an ML.NET pipeline
- **Execution providers** — CPU (default), CUDA GPU, DirectML, CoreML, selecting and configuring providers
- **Model sources** — ONNX Model Zoo, Hugging Face ONNX exports, custom Python→ONNX exports
- **Tensor I/O** — `references/tensors.md` for shape management, named inputs/outputs, batch inference
**Does NOT cover:**
- Training models from scratch (use `mlnet` for classical ML and supported deep learning; TorchSharp for custom architectures)
- LLM text generation (use `meai-chat-integration` or `local-llm-inference`)
### local-llm-inference
Guides developers through running LLMs locally via **Ollama** and **Foundry Local**, both surfacing models through MEAI's `IChatClient` abstraction.
**Covers:**
- **Ollama** — installation, pulling models, connecting via `OllamaSharp` as IChatClient, model selection
- **Foundry Local** — installation, `Microsoft.AI.Foundry.Local` SDK, launching models, connecting as IChatClient
- **Provider-agnostic patterns** — both produce IChatClient, so all MEAI middleware (streaming, function calling, caching) works identically
- **Model selection** — guidance on choosing models by task (chat, code, small/large)
**Does NOT cover:**
- Cloud-hosted LLMs (use `meai-chat-integration`)
- Non-LLM models (use `onnx-runtime-inference`)
- Classical ML tasks (use `mlnet`)
## Reference Files
| File | Purpose |
|---|---|
| `references/tensors.md` | Tensor I/O patterns for ONNX Runtime — shape management, named inputs/outputs, batch processing, memory management with `OrtValue` |
## Files
```
plugins/dotnet-ai/skills/onnx-runtime-inference/
SKILL.md
references/
tensors.md
plugins/dotnet-ai/skills/local-llm-inference/SKILL.md
tests/dotnet-ai/onnx-runtime-inference/eval.yaml
tests/dotnet-ai/local-llm-inference/eval.yaml
```
## Eval Scenarios
**onnx-runtime-inference:**
1. **Run ONNX model for inference** — prompt asks to run a pre-trained image classification model; expects InferenceSession or ApplyOnnxModel with execution provider selection
**local-llm-inference:**
1. **Set up Ollama for local development** — prompt asks to run an LLM locally without API keys; expects Ollama setup with IChatClient
2. **Reject local inference for production cloud** — prompt asks to use local inference for a cloud-deployed production app; expects redirect to cloud providers
## Key Packages
- `Microsoft.ML.OnnxRuntime` (standalone ONNX)
- `Microsoft.ML.OnnxRuntime.Gpu` (CUDA)
- `Microsoft.ML.OnnxRuntime.DirectML` (DirectML)
- `Microsoft.ML.OnnxTransformer` (ML.NET integration)
- `OllamaSharp` (Ollama IChatClient)
- `Microsoft.AI.Foundry.Local` (Foundry Local)
## Dependencies
Requires the plugin scaffold (sub-issue #1) to be merged first.
## Acceptance Criteria
- [ ] onnx-runtime-inference covers both standalone and ML.NET approaches
- [ ] Execution providers (CPU, CUDA, DirectML) are documented
- [ ] tensors.md reference provides tensor shape and memory management guidance
- [ ] local-llm-inference presents Ollama and Foundry Local neutrally (no prioritization)
- [ ] Both skills surface models through IChatClient abstraction
- [ ] eval.yaml files have scenarios for each skill
Contributor guide
Assessment
This issue has not been assessed yet.