apple / apple/coremltools

Can the decode of transformer be accelerated by NPU

Open
#2,612 2 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🌱 Describe Feature Request
I trained a Transformer model. When I converted it as a whole into an mlmodel, I found that its intelligence could only be processed on the cpu. After splitting it into encode and decode, I discovered that encode could be normally accelerated using the NPU, but decode could only be processed on the cpu. Is it because decode is self-decoding, not a static issue? If decode can be accelerated by NPU, could a method for converting pt to mlmodel or mlpackage be provided
thanks

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reproducing the reported Transformer conversion with encode and decode separated, then compare CPU and NPU processing; done would require a documented conversion path from pt to mlmodel or mlpackage that supports NPU decode, if supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.