dotnet / dotnet/machinelearning

Improve ONNX inference performance by adopting the OrtValue API

Open
#7,682 1 comment 0 reactions 0 assignees View on GitHub
area-ONNX tenet-performance
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

`OnnxTransformer` still invokes ONNX Runtime through the legacy `NamedOnnxValue` API. ONNX Runtime now recommends its `OrtValue` API, which produces less garbage, supports reusable input and output buffers, and can improve inference performance.

ML.NET currently references ONNX Runtime 1.23.2. The required C# API was introduced in 1.16, so upgrading ONNX Runtime is not a prerequisite for this work.

## Motivation

The current inference path:

- Creates `NamedOnnxValue` inputs for every row.
- Copies numeric inputs using `data.ToArray()`.
- Creates intermediate `string[]` and `DenseTensor` instances for string inputs.
- Lets ONNX Runtime allocate output values for every invocation.
- Converts and copies results into ML.NET `VBuffer` instances.
- Performs one ONNX Runtime invocation per `IDataView` row.

These allocations can increase GC pressure and latency, particularly for small models and variable-length string inputs. This is related to #6620.

ONNX Runtime describes `NamedOnnxValue` as a legacy API and reports that `OrtValue` can provide substantial performance and allocation improvements in some scenarios.

## Proposal

Benchmark and, where beneficial, migrate `OnnxTransformer` to the `OrtValue` API:

1. Use `OrtValue.CreateTensorValueFromMemory` with reusable managed buffers for numeric inputs.
2. Preallocate and reuse numeric output tensors when their shapes are fixed.
3. Access dynamically allocated outputs through `GetTensorDataAsSpan()`.
4. Populate native string tensors directly from ML.NET values, avoiding intermediate `string[]` and `DenseTensor` allocations where possible.
5. Reuse input names, output names, shapes, and `RunOptions` rather than recreating them for every row.
6. Dispose native values deterministically and ensure managed buffers are not left pinned.
7. Preserve the existing behaviour for dynamic shapes, strings, maps, sequences, and other supported ONNX value types.

String tensors will still require UTF-16 to UTF-8 conversion and copying, but the new API should allow us to remove several intermediate managed allocations.

## Benchmark plan

Compare the existing and `OrtValue` paths using ONNX Runtime 1.23.2 first, so the effect of the API migration can be measured independently.

Cover at least:

- Fixed-shape numeric inputs and outputs.
- Dynamic-shape numeric inputs or outputs.
- Fixed and variable-length string inputs.
- Single and multiple input/output models.
- Small models where managed overhead is significant.
- Longer-running models to check for regressions.

Measure:

- Mean inference latency and throughput.
- Allocated bytes per inference.
- Gen 0, Gen 1, and Gen 2 collections.
- Performance under repeated scoring.
- Memory and handle stability over prolonged execution.

The latest stable ONNX Runtime version should be evaluated separately. Any dependency update should ideally be made in a separate change so its kernel and execution-provider improvements can be measured independently.

## Acceptance criteria

- Benchmarks document the performance and allocation differences.
- The `OrtValue` path is used where it improves performance without breaking existing behaviour.
- Reusable buffers are used for fixed-shape numeric tensors.
- Intermediate string arrays and tensors are removed where the API permits.
- Existing ONNX transformer tests continue to pass.
- Dynamic shapes and all currently supported input and output types remain compatible.
- Repeated inference does not leak native memory or leave managed buffers pinned.

## Out of scope

- Changing `IDataView` to perform automatic batching.
- Introducing device-backed `VBuffer` values.
- Keeping complete pipelines resident in GPU memory.

Those changes may provide larger GPU throughput improvements, but require broader ML.NET API and pipeline design work. `OrtIoBinding` already has an official C# binding and can be investigated separately for such scenarios.

## References

- [ONNX Runtime C# OrtValue guidance](https://onnxruntime.ai/docs/tutorials/csharp/basic_csharp.html)
- [ONNX Runtime C# OrtValue implementation](https://github.com/microsoft/onnxruntime/pull/16206)
- [NamedOnnxValue API documentation](https://onnxruntime.ai/docs/api/csharp/api/Microsoft.ML.OnnxRuntime.NamedOnnxValue.html)
- [ML.NET GC pressure report](https://github.com/dotnet/machinelearning/issues/6620)
- [Related ONNX Runtime investigation](https://github.com/microsoft/onnxruntime/issues/15488)

Contributor guide

Open the contributing guide

Research direction

Start by reading the OnnxTransformer inference path and running the existing ONNX transformer tests. Benchmark the current NamedOnnxValue path against an OrtValue path with ONNX Runtime 1.23.2 across the listed input and output cases. Done means documented performance results, preserved supported behavior, and no native-memory leaks or lingering pinned buffers.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
machine-learning, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.