How to select 128 samples for MASQuant
- Dominant language
- Python
- Stars
- 51
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
Hi authors,
I noticed that the calibration data preparation selects the first 128 samples from COCO:
[custom_dataset.py#L101](https://github.com/alibaba/EfficientAI/blob/3d32ae427eec57166ea67f3018cd4568be84496f/masquant/custom_dataset.py#L101)
During my reproduction, I found that the choice of these 128 calibration samples has a noticeable impact on the final performance. When I use different subsets of 128 samples, the evaluation results can vary by around 1-2%, which is comparable to the performance gap between several PTQ methods reported in Table 1. And the fluctuation (caused by 128 samples) happens on all PTQ methods.
Can you help me fix the issue?
- How do you report it, e.g. avg over 3 runs?
- If you use a fixed set of 128 samples, could you upload the JSON file?
- Due to the fluctuation, how to fairly compare MASQuant with MBQ and AWQ?
Contributor guide
No contributing guide indexed for this repository
Research direction
Inspect masquant/custom_dataset.py at line 101 to understand how the first 128 COCO samples are selected, then reproduce the reported variation with different 128-sample subsets. Done means the calibration set and evaluation procedure are clarified, including whether results are averaged over runs and whether a fixed JSON sample list is provided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100