apple / apple/coremltools

The NPU cannot accelerate normally

Open
#2,564 2 comments 1 reaction 0 assignees View on GitHub
question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓softmax operations cannot be accelerated by NPU
I trained a character recognition model. The image encoding part adopted the encode of transformer. During decoding, I only used one linear layer and softmax operation. When only the linear layer was output, the model could be accelerated normally, but when softmax was added, it could not be accelerated normally.

Three methods were used for softmax operation, but none of them were feasible
F.log_softmax()
torch.softmax()
output.softmax()

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the character-recognition model conversion and compare NPU acceleration with only the linear layer versus each of F.log_softmax(), torch.softmax(), and output.softmax(). Inspect the conversion and acceleration behavior for the softmax operation. Done means the model's softmax path accelerates normally, or the unsupported limitation is documented with a reproducible explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.