huggingface / huggingface/datasets
Concurrent use of same dataset (already downloaded)
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
## Describe the bug
When launching several jobs at the same time loading the same dataset trigger some errors see (last comments).
## Steps to reproduce the bug
export HF_DATASETS_CACHE=/gpfswork/rech/toto/datasets
for MODEL in "bert-base-uncased" "roberta-base" "distilbert-base-cased"; do # "bert-base-uncased" "bert-large-cased" "roberta-large" "albert-base-v1" "albert-large-v1"; do
for TASK_NAME in "mrpc" "rte" 'imdb' "paws" "mnli"; do
export OUTPUT_DIR=${MODEL}_${TASK_NAME}
sbatch --job-name=${OUTPUT_DIR} \
--gres=gpu:1 \
--no-requeue \
--cpus-per-task=10 \
--hint=nomultithread \
--time=1:00:00 \
--output=jobinfo/${OUTPUT_DIR}_%j.out \
--error=jobinfo/${OUTPUT_DIR}_%j.err \
--qos=qos_gpu-t4 \
--wrap="module purge; module load pytorch-gpu/py3/1.7.0 ; export HF_DATASETS_OFFLINE=1; export HF_DATASETS_CACHE=/gpfswork/rech/toto/datasets; python compute_measures.py --seed=$SEED --saving_path=results --batch_size=$BATCH_SIZE --task_name=$TASK_NAME --model_name=/gpfswork/rech/toto/transformers_models/$MODEL"
done
done
```python
# Sample code to reproduce the bug
dataset_train = load_dataset('imdb', split='train', download_mode="reuse_cache_if_exists")
dataset_train = dataset_train.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
batched=True).select(list(range(args.filter)))
dataset_val = load_dataset('imdb', split='train', download_mode="reuse_cache_if_exists")
dataset_val = dataset_val.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
batched=True).select(list(range(args.filter, args.filter + 5000)))
dataset_test = load_dataset('imdb', split='test', download_mode="reuse_cache_if_exists")
dataset_test = dataset_test.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
batched=True)
```
## Expected results
I believe I am doing something wrong with the objects.
## Actual results
Traceback (most recent call last):
File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 983, in _prepare_split
check_duplicates=True,
File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/arrow_writer.py", line 192, in __init__
self.stream = pa.OSFile(self._path, "wb")
File "pyarrow/io.pxi", line 829, in pyarrow.lib.OSFile.__cinit__
File "pyarrow/io.pxi", line 844, in pyarrow.lib.OSFile._open_writable
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
FileNotFoundError: [Errno 2] Failed to open local file '/gpfswork/rech/tts/unm25jp/datasets/paws/labeled_final/1.1.0/09d8fae989bb569009a8f5b879ccf2924d3e5cd55bfe2e89e6dab1c0b50ecd34.incomplete/paws-test.arrow'. Detail: [errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "compute_measures.py", line 181, in
train_loader, val_loader, test_loader = get_dataloader(args)
File "/gpfsdswork/projects/rech/toto/intRAOcular/dataset_utils.py", line 69, in get_dataloader
dataset_train = load_dataset('paws', "labeled_final", split='train', download_mode="reuse_cache_if_exists")
File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
use_auth_token=use_auth_token,
File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 658, in _download_and_prepare
+ str(e)
OSError: Cannot find data file.
Original error:
[Errno 2] Failed to open local file '/gpfswork/rech/toto/datasets/paws/labeled_final/1.1.0/09d8fae989bb569009a8f5b879ccf2924d3e5cd55bfe2e89e6dab1c0b50ecd34.incomplete/paws-test.arrow'. Detail: [errno 2] No such file or directory
## Environment info
- `datasets` version: datasets==1.8.0
- Platform: linux (jeanzay)
- Python version: pyarrow==2.0.0
- PyArrow version: 3.7.8
Contributor guide
Assessment
This issue has not been assessed yet.