dotnet / dotnet/skills

[dotnet-ai] ONNX Runtime + local LLM inference

Open
#231 0 comments 0 reactions 0 assignees View on GitHub
area-ai skill
Dominant language
C#
Stars
5.4k
Forks
415
Avg merge
1d 4h
Merged PRs (30d)
84

Description

## Parent Issue

Part of the `dotnet-ai` plugin proposal: #225

## Summary

Add the `onnx-runtime-inference` and `local-llm-inference` skills to the `dotnet-ai` plugin, including one reference file.

These two skills cover model execution — running pre-trained models and LLMs locally. They are submitted together because they share the deployment-at-the-edge theme and reference each other in their routing guidance.

## Skills in This Issue

### onnx-runtime-inference

Guides developers through running pre-trained ONNX models for inference in .NET, both standalone (via `Microsoft.ML.OnnxRuntime`) and through ML.NET (via `Microsoft.ML.OnnxTransformer`).

**Covers:**
- **Standalone approach** — `InferenceSession`, input/output tensor construction, execution
- **ML.NET approach** — `ApplyOnnxModel()` within an ML.NET pipeline
- **Execution providers** — CPU (default), CUDA GPU, DirectML, CoreML, selecting and configuring providers
- **Model sources** — ONNX Model Zoo, Hugging Face ONNX exports, custom Python→ONNX exports
- **Tensor I/O** — `references/tensors.md` for shape management, named inputs/outputs, batch inference

**Does NOT cover:**
- Training models from scratch (use `mlnet` for classical ML and supported deep learning; TorchSharp for custom architectures)
- LLM text generation (use `meai-chat-integration` or `local-llm-inference`)

### local-llm-inference

Guides developers through running LLMs locally via **Ollama** and **Foundry Local**, both surfacing models through MEAI's `IChatClient` abstraction.

**Covers:**
- **Ollama** — installation, pulling models, connecting via `OllamaSharp` as IChatClient, model selection
- **Foundry Local** — installation, `Microsoft.AI.Foundry.Local` SDK, launching models, connecting as IChatClient
- **Provider-agnostic patterns** — both produce IChatClient, so all MEAI middleware (streaming, function calling, caching) works identically
- **Model selection** — guidance on choosing models by task (chat, code, small/large)

**Does NOT cover:**
- Cloud-hosted LLMs (use `meai-chat-integration`)
- Non-LLM models (use `onnx-runtime-inference`)
- Classical ML tasks (use `mlnet`)

## Reference Files

| File | Purpose |
|---|---|
| `references/tensors.md` | Tensor I/O patterns for ONNX Runtime — shape management, named inputs/outputs, batch processing, memory management with `OrtValue` |

## Files

```
plugins/dotnet-ai/skills/onnx-runtime-inference/
SKILL.md
references/
tensors.md
plugins/dotnet-ai/skills/local-llm-inference/SKILL.md
tests/dotnet-ai/onnx-runtime-inference/eval.yaml
tests/dotnet-ai/local-llm-inference/eval.yaml
```

## Eval Scenarios

**onnx-runtime-inference:**
1. **Run ONNX model for inference** — prompt asks to run a pre-trained image classification model; expects InferenceSession or ApplyOnnxModel with execution provider selection

**local-llm-inference:**
1. **Set up Ollama for local development** — prompt asks to run an LLM locally without API keys; expects Ollama setup with IChatClient
2. **Reject local inference for production cloud** — prompt asks to use local inference for a cloud-deployed production app; expects redirect to cloud providers

## Key Packages

- `Microsoft.ML.OnnxRuntime` (standalone ONNX)
- `Microsoft.ML.OnnxRuntime.Gpu` (CUDA)
- `Microsoft.ML.OnnxRuntime.DirectML` (DirectML)
- `Microsoft.ML.OnnxTransformer` (ML.NET integration)
- `OllamaSharp` (Ollama IChatClient)
- `Microsoft.AI.Foundry.Local` (Foundry Local)

## Dependencies

Requires the plugin scaffold (sub-issue #1) to be merged first.

## Acceptance Criteria

- [ ] onnx-runtime-inference covers both standalone and ML.NET approaches
- [ ] Execution providers (CPU, CUDA, DirectML) are documented
- [ ] tensors.md reference provides tensor shape and memory management guidance
- [ ] local-llm-inference presents Ollama and Foundry Local neutrally (no prioritization)
- [ ] Both skills surface models through IChatClient abstraction
- [ ] eval.yaml files have scenarios for each skill

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.