huggingface / huggingface/datasets
MMLU get_dataset_config_names provides different lists of subsets in online and offline modes
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
When getting the config names of `cais/mmlu` in online mode, it provides the different subjects and `all` but in offline mode is provides `default` even with cached version of the dataset.
### Steps to reproduce the bug
1. First download dataset in online mode so that it is cached:
```
$ HF_DATASETS_OFFLINE=0 HF_HOME="/tmp/hftemp" python -c "import datasets;print(datasets.get_dataset_config_names('cais/mmlu'));datasets.load_dataset('cais/mmlu', 'all')"
['abstract_algebra', 'all', 'anatomy', 'astronomy', 'auxiliary_train', 'business_ethics', 'clinical_knowledge', 'college_biology', 'college_chemistry', 'college_computer_science', 'college_mathematics', 'college_medicine', 'college_physics', 'computer_security', 'conceptual_physics', 'econometrics', 'electrical_engineering', 'elementary_mathematics', 'formal_logic', 'global_facts', 'high_school_biology', 'high_school_chemistry', 'high_school_computer_science', 'high_school_european_history', 'high_school_geography', 'high_school_government_and_politics', 'high_school_macroeconomics', 'high_school_mathematics', 'high_school_microeconomics', 'high_school_physics', 'high_school_psychology', 'high_school_statistics', 'high_school_us_history', 'high_school_world_history', 'human_aging', 'human_sexuality', 'international_law', 'jurisprudence', 'logical_fallacies', 'machine_learning', 'management', 'marketing', 'medical_genetics', 'miscellaneous', 'moral_disputes', 'moral_scenarios', 'nutrition', 'philosophy', 'prehistory', 'professional_accounting', 'professional_law', 'professional_medicine', 'professional_psychology', 'public_relations', 'security_studies', 'sociology', 'us_foreign_policy', 'virology', 'world_religions']
all/test-00000-of-00001.parquet: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.50M/3.50M [00:00<00:00, 3.78MB/s]
all/validation-00000-of-00001.parquet: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 408k/408k [00:00<00:00, 1.90MB/s]
all/dev-00000-of-00001.parquet: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76.5k/76.5k [00:00<00:00, 298kB/s]
all/auxiliary_train-00000-of-00001.parqu(…): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 47.5M/47.5M [00:00<00:00, 47.6MB/s]
Generating test split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:01<00:00, 10914.95 examples/s]
Generating validation split: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1531/1531 [00:00<00:00, 271131.54 examples/s]
Generating dev split: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 285/285 [00:00<00:00, 86308.78 examples/s]
Generating auxiliary_train split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 99842/99842 [00:00<00:00, 202836.59 examples/s]
```
2. Verify the config names of in offline mode (the dataset has been cached).
```
$ HF_DATASETS_OFFLINE=1 HF_HOME="/tmp/hftemp" python -c "import datasets;print(datasets.get_dataset_config_names('cais/mmlu'))"
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub (offline mode is enabled).
['default']
```
### Expected behavior
It should return the same list of configs in both cases.
### Environment info
$ python -c "import datasets;print(datasets.__version__)"
4.4.1
Contributor guide
Assessment
This issue has not been assessed yet.