alibaba / alibaba/EfficientAI

How to select 128 samples for MASQuant

Open
#9 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
51
Forks
8
PR merge metrics
No merged PRs in 30d

Description

Hi authors,

I noticed that the calibration data preparation selects the first 128 samples from COCO:
[custom_dataset.py#L101](https://github.com/alibaba/EfficientAI/blob/3d32ae427eec57166ea67f3018cd4568be84496f/masquant/custom_dataset.py#L101)

During my reproduction, I found that the choice of these 128 calibration samples has a noticeable impact on the final performance. When I use different subsets of 128 samples, the evaluation results can vary by around 1-2%, which is comparable to the performance gap between several PTQ methods reported in Table 1. And the fluctuation (caused by 128 samples) happens on all PTQ methods.

Can you help me fix the issue?
- How do you report it, e.g. avg over 3 runs?
- If you use a fixed set of 128 samples, could you upload the JSON file?
- Due to the fluctuation, how to fairly compare MASQuant with MBQ and AWQ?

Contributor guide

No contributing guide indexed for this repository

Research direction

Inspect masquant/custom_dataset.py at line 101 to understand how the first 128 COCO samples are selected, then reproduce the reported variation with different 128-sample subsets. Done means the calibration set and evaluation procedure are clarified, including whether results are averaged over runs and whether a fixed JSON sample list is provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.