microsoft / microsoft/onnxruntime
[Feature Request] Support BERT models In CoreML execution provider
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
Describe the feature request
We want to run on-device a fine-tuned BERT model for token classification.
Currently, the partial support of the models' operations does not utilize ANE/GPU fully, and some operations are assigned to the CPU, causing the model to be too slow for production purposes.
Converting the model to .mlpackage and running it directly with CoreML works with ANE/GPU and runs fast.
It seems there are 2 parts to enable BERT models support:
- Support
dim > 16384- base BERT default vocab size is30522(source), which requires retraining the model with a smaller, more limited vocab size - Support the relevant operations. When loading the model, this are the operations assigned to the CPU:
2025-07-16 13:16:26.883836 [V:onnxruntime:, session_state.cc:1266 VerifyEachNodeIsAssignedToAnEp] Node(s) placed on [CPUExecutionProvider]. Number of nodes: 12
2025-07-16 13:16:26.883841 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] Unsqueeze (/bert/Unsqueeze)
2025-07-16 13:16:26.883845 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] Unsqueeze (/bert/Unsqueeze_1)
2025-07-16 13:16:26.883850 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] Expand (/bert/Expand)
2025-07-16 13:16:26.883854 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] Cast (/bert/Cast)
2025-07-16 13:16:26.883858 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] Cast (/bert/Cast_1)
2025-07-16 13:16:26.883863 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] Where (/bert/Where_1)
2025-07-16 13:16:26.883867 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] MatMul (/bert/encoder/layer.0/attention/self/MatMul)
2025-07-16 13:16:26.883872 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] MatMul (/bert/encoder/layer.0/attention/self/MatMul_1)
2025-07-16 13:16:26.883876 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] MatMul (/bert/encoder/layer.1/attention/self/MatMul)
2025-07-16 13:16:26.883880 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] MatMul (/bert/encoder/layer.1/attention/self/MatMul_1)
2025-07-16 13:16:26.883885 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] MatMul (/bert/encoder/layer.2/attention/self/MatMul)
2025-07-16 13:16:26.883892 [V:onnxruntime:, session_state.cc:1268 VerifyEachNodeIsAssignedToAnEp] MatMul (/bert/encoder/layer.2/attention/self/MatMul_1)
Describe scenario use case
BERT is a very common model family used for natural text tasks.
While GPT-like models dominate a lot of text tasks, specifically classification tasks on-device will probably be left for the BERT family and it's evolutions (modern-BERT) - they are much smaller in size, and better suited for classification as an encoder.
Supporting them in ONNX will allow also efficient cross-platform hardware utilization.
They are already supported in OpenVino and Cuda Execution Providers.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the CoreML execution provider and the node-assignment output described in the issue. Identify how the dim limit and the listed Unsqueeze, Expand, Cast, Where, and MatMul operations are handled, then verify that BERT token-classification models use ANE/GPU rather than CPU for these operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- machine-learning
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100