microsoft / microsoft/onnxruntime

GeluFusion emits com.microsoft.Gelu(float16) on linux/arm64 where no float16 kernel is registered, so fp16 Erf-GELU models fail to load

Open
#32,386 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

On linux/aarch64 a plain opset 17 model containing the standard Erf based GELU pattern in float16 fails to load. The graph optimizer fuses the pattern into `com.microsoft.Gelu`, and then the same build reports it has no float16 kernel for the node it just created.

```
[ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for com.microsoft.Gelu(1)
(node:'Gelu' ep:'CPUExecutionProvider'). ... This op has been implemented only for the
following types (tensor(float),), but the node in the model has the following type
(tensor(float16))
```

The model on disk contains no contrib ops at all. The node named in the error does not exist in the file. It is created during session load, which is what makes this confusing to debug: grepping the model for `com.microsoft.Gelu` finds nothing.

The same model, the same wheel version, and the same providers list loads fine on linux/x86_64.

This is a regression. It loads on 1.23.0 and 1.23.2, and fails on 1.24.2 and on 1.29.0. I could not test 1.24.0 or 1.24.1 because there are no aarch64 wheels for them on PyPI.

### Urgency

Moderate. There is a workaround, but the failure mode is unpleasant. Any fp16 model using Erf GELU, which covers a lot of exported transformer graphs, stops loading on arm64 while continuing to work on x86_64, and the error names an operator the user never put in the model. I hit it on an exported wav2vec2 classifier where the result was a feature that silently disappeared on one platform.

### To reproduce

Generates the model. Five nodes, about 1 KB, no external weights, nothing downloaded.

```python
import numpy as np, onnx
from onnx import helper, TensorProto, numpy_helper

f16 = TensorProto.FLOAT16
c = lambda n, v: numpy_helper.from_array(np.array(v, dtype=np.float16), n)

g = helper.make_graph(
[
helper.make_node("Div", ["X", "sqrt2"], ["div"]),
helper.make_node("Erf", ["div"], ["erf"]),
helper.make_node("Add", ["erf", "one"], ["add"]),
helper.make_node("Mul", ["X", "add"], ["mul"]),
helper.make_node("Mul", ["mul", "half"], ["Y"]),
],
"erf_gelu_fp16",
[helper.make_tensor_value_info("X", f16, [1, 8])],
[helper.make_tensor_value_info("Y", f16, [1, 8])],
[c("sqrt2", 1.4142135), c("one", 1.0), c("half", 0.5)],
)
m = helper.make_model(g, opset_imports=[helper.make_opsetid("", 17)])
m.ir_version = 9
onnx.checker.check_model(m)
onnx.save(m, "erf_gelu_fp16.onnx")
```

Loads it.

```python
import onnxruntime as ort

so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
ort.InferenceSession("erf_gelu_fp16.onnx", so, providers=["CPUExecutionProvider"])
```

Run on a native aarch64 machine, not under emulation:

```
docker run --rm python:3.12-slim bash -c "pip -q install onnx numpy onnxruntime && ..."
```

### Results

Everything below is on native hardware, stock PyPI wheels, `CPUExecutionProvider` only. The fp32 control is the identical graph with float32 initializers and I/O.

| platform | version | graph | opt level | result |
| --- | --- | --- | --- | --- |
| linux/aarch64 | 1.24.2 | fp16 | ORT_ENABLE_ALL | **fails to load** |
| linux/aarch64 | 1.29.0 | fp16 | ORT_ENABLE_ALL | **fails to load** |
| linux/aarch64 | 1.23.0 | fp16 | ORT_ENABLE_ALL | loads |
| linux/aarch64 | 1.23.2 | fp16 | ORT_ENABLE_ALL | loads |
| linux/aarch64 | 1.24.2 | fp16 | ORT_ENABLE_BASIC | loads |
| linux/aarch64 | 1.24.2 | fp32 | ORT_ENABLE_ALL | loads |
| linux/x86_64 | 1.24.2 | fp16 | ORT_ENABLE_ALL | loads |
| linux/x86_64 | 1.29.0 | fp16 | ORT_ENABLE_ALL | loads |

`ORT_ENABLE_BASIC` loading, and `ORT_ENABLE_ALL` on the fp32 graph loading, together place this squarely in the level 2 fusion rather than anywhere in kernel dispatch for the ops the user actually wrote.

Setting `optimization.disable_specified_optimizers` to `GeluFusionL2` also makes it load, and on my real model the numerics were unchanged to within 9.6e-04 max absolute logit difference against the unoptimized reference, with identical labels.

### Expected behavior

The optimizer should not emit a node that the same build has no kernel for. Either register a float16 `Gelu` kernel for the aarch64 CPU EP, or have `GeluFusion` check kernel availability for the fused node's element type before rewriting and leave the graph alone when there is none.

The second option seems more generally useful. Any type driven kernel gap in a fusion target has this same shape, and today it turns into a load failure that names an operator the user never wrote.

### Secondary observation

macOS arm64 does not hit this. I checked the static 1.24.2 library the Rust `ort` crate links there, and it has the same gap, one `Gelu` instantiation and none for `MLFloat16`, but the fusion does not fire on fp16 in that build, so the session opens. Two arm64 targets of the same release therefore disagree about whether the rewrite happens. That is a separate measurement path from the pip wheels above, so treat it as a lead rather than as part of the report.

### Platform

Linux

### OS Version

Ubuntu 24.04 (aarch64 and x86_64)

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.24.2, 1.29.0 (fail); 1.23.0, 1.23.2 (load)

### ONNX Runtime API

Python

### Architecture

ARM64

### Execution Provider

Default CPU

Contributor guide

Open the contributing guide

Research direction

Reproduce the provided five-node fp16 model on native linux/aarch64 with ORT_ENABLE_ALL, then trace the GeluFusionL2 graph optimizer and the CPU execution provider's kernel availability. Compare the fusion path with ORT_ENABLE_BASIC and the disabled-optimizer workaround. Done means the optimizer no longer produces an unloadable com.microsoft.Gelu node, while the existing fp32 and x86_64 cases remain working.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.