huggingface / huggingface/datasets

AttributeError: type object 'tqdm' has no attribute '_lock'

Open
#7,660 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

`AttributeError: type object 'tqdm' has no attribute '_lock'`

It occurs when I'm trying to load datasets in thread pool.

Issue https://github.com/huggingface/datasets/issues/6066 and PR https://github.com/huggingface/datasets/pull/6067 https://github.com/huggingface/datasets/pull/6068 tried to fix this.

### Steps to reproduce the bug

Will have to try several times to reproduce the error because it is concerned with threads.

1. save some datasets for test
```pythonfrom datasets import Dataset, DatasetDict
import os

os.makedirs("test_dataset_shards", exist_ok=True)

for i in range(10):
data = Dataset.from_dict({"text": [f"example {j}" for j in range(1000000)]})
data = DatasetDict({'train': data})
data.save_to_disk(f"test_dataset_shards/shard_{i}")
```

2. load them in a thread pool
```python
from datasets import load_from_disk
from tqdm import tqdm
from concurrent.futures import ThreadPoolExecutor, as_completed
import glob

datas = glob.glob('test_dataset_shards/shard_*')
with ThreadPoolExecutor(max_workers=10) as pool:
futures = [pool.submit(load_from_disk, it) for it in datas]
datas = []
for future in tqdm(as_completed(futures), total=len(futures)):
datas.append(future.result())
```

### Expected behavior

no exception raised

### Environment info

datasets==2.19.0
python==3.10

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.