apple / apple/coremltools

need help about both model weight and activation quantization with only a float32 mlmodel

Open
#2,227 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

from the issue "https://developer.apple.com/forums/thread/740518 how do we use the computational power of A17 Pro Neural Engine?"

I learn that if i want to inference my mlmodel on my ipad pro with m4 soc int8 38T ane high performance, i have to use the coreml torch api to quantize both weight and activation during training time quantization with int8 datatype.

my question is:
I only have a fp32 mlmodel without torch code or model, what can i do?
by the way, if just only weight int8 quantization, M4 ane will use fp16 to compute or int8?
thanks for your help~

Contributor guide

Open the contributing guide

Research direction

Start with the linked Apple Developer forum discussion and the coremltools quantization documentation or APIs. Determine whether an existing float32 .mlmodel can be quantized without the original training code, and clarify the execution behavior of weight-only int8 quantization on the M4 Neural Engine. Done means providing a documented, version-specific answer or identifying the missing prerequisite.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, tooling
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.