google-research / google-research/google-research

Cannot replicate results table 1 from Distilling Effective Supervision from Severe Label Noise

Open
#702 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
38.8k
Forks
8.5k
PR merge metrics
No merged PRs in 30d

Description

I clone repository at https://github.com/google-research/google-research/tree/master/ieg , set 'use_imagenet_as_eval' to False
then I run command:
CUDA_VISIBLE_DEVICES=0 python -m ieg.main --dataset=cifar10_uniform_0.2 --network_name=resnet29 --probe_dataset_hold_ratio=0.002 --checkpoint_path=${SAVEPATH}/ieg

That's all step, but in paper the results is ~0.92, when I test it is only 0.8x.

I use 1 GPU cuda 10.1 cudnn 7.6.5 with all required package in conda env.

Contributor guide

Open the contributing guide

Research direction

Start in the ieg repository and rerun the reported command with use_imagenet_as_eval set to False, the cifar10_uniform_0.2 dataset, resnet29, and the stated probe ratio. Compare the resulting accuracy with the paper's approximately 0.92 result and check the listed CUDA, cuDNN, and package environment. Done means reproducing the discrepancy and identifying its cause or documenting the required setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.