huggingface / huggingface/datasets

Loading a dataset from the hub fails if you happen to have a folder of the same name

Open
#5,228 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

I'm not 100% sure this should be considered a bug, but it was certainly annoying to figure out the cause of. And perhaps I am just missing a specific argument needed to avoid this conflict. Basically I had a situation where multiple workers were downloading different parts of the glue dataset and then training on them. Additionally, they were writing their checkpoints to a folder called `glue`. This meant that once one worker had created the `glue` folder to write checkpoints to, the next worker to try to load a glue dataset would fail as shown in the minimal repro below. I'm not sure what the solution would be since I'm not super familiar with the `datasets` code, but I would expect `load_dataset` to not crash just because i have a local folder with the same name as a dataset from the hub.

### Steps to reproduce the bug

```
In [1]: import datasets

In [2]: rte = datasets.load_dataset('glue', 'rte')
Downloading and preparing dataset glue/rte to /Users/danielking/.cache/huggingface/datasets/glue/rte/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad...
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 697k/697k [00:00<00:00, 6.08MB/s]
Dataset glue downloaded and prepared to /Users/danielking/.cache/huggingface/datasets/glue/rte/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 773.81it/s]

In [3]: import os

In [4]: os.mkdir('glue')

In [5]: rte = datasets.load_dataset('glue', 'rte')
---------------------------------------------------------------------------
EmptyDatasetError Traceback (most recent call last)
in ()
----> 1 rte = datasets.load_dataset('glue', 'rte')

~/miniconda3/envs/composer/lib/python3.9/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
1717
1718 # Create a dataset builder
-> 1719 builder_instance = load_dataset_builder(
1720 path=path,
1721 name=name,

~/miniconda3/envs/composer/lib/python3.9/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)
1495 download_config = download_config.copy() if download_config else DownloadConfig()
1496 download_config.use_auth_token = use_auth_token
-> 1497 dataset_module = dataset_module_factory(
1498 path,
1499 revision=revision,

~/miniconda3/envs/composer/lib/python3.9/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
1152 ).get_module()
1153 elif os.path.isdir(path):
-> 1154 return LocalDatasetModuleFactoryWithoutScript(
1155 path, data_dir=data_dir, data_files=data_files, download_mode=download_mode
1156 ).get_module()

~/miniconda3/envs/composer/lib/python3.9/site-packages/datasets/load.py in get_module(self)
624 base_path = os.path.join(self.path, self.data_dir) if self.data_dir else self.path
625 patterns = (
--> 626 sanitize_patterns(self.data_files) if self.data_files is not None else get_data_patterns_locally(base_path)
627 )
628 data_files = DataFilesDict.from_local_or_remote(

~/miniconda3/envs/composer/lib/python3.9/site-packages/datasets/data_files.py in get_data_patterns_locally(base_path)
458 return _get_data_files_patterns(resolver)
459 except FileNotFoundError:
--> 460 raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None
461
462

EmptyDatasetError: The directory at glue doesn't contain any data files
```

### Expected behavior

Dataset is still able to be loaded from the hub even if I have a local folder with the same name.

### Environment info

datasets version: 2.6.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.