google-research / google-research/google-research
Cannot replicate results table 1 from Distilling Effective Supervision from Severe Label Noise
- Dominant language
- Jupyter Notebook
- Stars
- 38.8k
- Forks
- 8.5k
- PR merge metrics
- No merged PRs in 30d
Description
I clone repository at https://github.com/google-research/google-research/tree/master/ieg , set 'use_imagenet_as_eval' to False
then I run command:
CUDA_VISIBLE_DEVICES=0 python -m ieg.main --dataset=cifar10_uniform_0.2 --network_name=resnet29 --probe_dataset_hold_ratio=0.002 --checkpoint_path=${SAVEPATH}/ieg
That's all step, but in paper the results is ~0.92, when I test it is only 0.8x.
I use 1 GPU cuda 10.1 cudnn 7.6.5 with all required package in conda env.
Contributor guide
Research direction
Start in the ieg repository and rerun the reported command with use_imagenet_as_eval set to False, the cifar10_uniform_0.2 dataset, resnet29, and the stated probe ratio. Compare the resulting accuracy with the paper's approximately 0.92 result and check the listed CUDA, cuDNN, and package environment. Done means reproducing the discrepancy and identifying its cause or documenting the required setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100