apple / apple/coremltools

Results for Optimizing OPT Model seems unusually high for 4 bits per tensor

Open
#2,352 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

Looking at the data for `4 bits per tensor` in the results table for optimizing OPT model, the values for `c4 Perplexity` and `wikitext2 Perplexity` seem unusually high compared to the other configurations.
Are these values correct or there is some typo?

Code link here: https://github.com/apple/coremltools/blob/c3ea4cf56fef1176417246c1b85363417f3e713d/docs-guides/source/opt-opt1_3.md?plain=1#L74

![OPT-model-results](https://github.com/user-attachments/assets/e355ba09-9ce7-455b-ac17-bbcd7b2386ec)

Contributor guide

Open the contributing guide

Research direction

Open docs-guides/source/opt-opt1_3.md around line 74 and inspect the 4 bits per tensor row in the OPT results table. Compare the c4 Perplexity and wikitext2 Perplexity entries with the other configurations and verify whether they are intentional; done means the table is confirmed or the suspected typo is corrected.

Written by the indexing model from the issue text.

Assessment

Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.