huggingface / huggingface/datasets

Add a robustness benchmark dataset for vision

Open
#5,371 1 comment 2 reactions 1 assignee Claimed by @sayakpaul View on GitHub
dataset request
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Name

ImageNet-C

### Paper

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

### Data

https://github.com/hendrycks/robustness

### Motivation

It's a known fact that vision models are brittle when they meet with slightly corrupted and perturbed data. This is also correlated to the robustness aspects of vision models.

Researchers use different benchmark datasets to evaluate the robustness aspects of vision models. ImageNet-C is one of them.

Having this dataset in 🤗 Datasets would allow researchers to evaluate and study the robustness aspects of vision models. Since the metric associated with these evaluations is top-1 accuracy, researchers should be able to easily take advantage of the evaluation benchmarks on the Hub and perform comprehensive reporting.

ImageNet-C is a large dataset. Once it's in, it can act as a reference and we can also reach out to the authors of the other robustness benchmark datasets in vision, such as ObjectNet, WILDS, Metashift, etc. These datasets cater to different aspects. For example, ObjectNet is related to assessing how well a model performs under sub-population shifts.

Related thread: https://huggingface.slack.com/archives/C036H4A5U8Z/p1669994598060499

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.