apple / apple/coremltools

Is it impossible to accelerate the decode of the Transformer using an NPU

Open
#2,566 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓The decode of the Transformer cannot be accelerated using an NPU
When I converted encode and decode to mlmodel, encode was accelerated normally with NPU, while decode could only be accelerated with CPU. I verified which step had the problem respectively and found that the issue occurred when adding position embeddings to the sequence, which prevented the model from being accelerated. However, encode also uses the same position embedding for acceleration. I guess it might be because decode has three inputs when passed into the model, and two of the input dimensions are variable, which affects the model's use of NPU acceleration.
If possible, please help me solve this problem and tell me which operations of the model can be accelerated and which cannot. In my previous verification, split could not support NPU acceleration

Contributor guide

Open the contributing guide

Research direction

Reproduce the encode and decode conversions to compare their position-embedding inputs and variable dimensions. Then isolate the decode operations, including split, and determine which prevent NPU acceleration; done means identifying the cause and documenting the operations or input shapes that are and are not supported.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.