huggingface / huggingface/datasets
Dataset.from_dict() can't handle large dict
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I have 26,000,000 3-tuples. When I use Dataset.from_dict() to load, neither. py nor Jupiter notebook can run successfully. This is my code:
```
# len(example_data) is 26,000,000, 'diff' is a text
diff1_list = [example_data[i].texts[0] for i in range(len(example_data))]
diff2_list = [example_data[i].texts[1] for i in range(len(example_data))]
label_list = [example_data[i].label for i in range(len(example_data))]
embedding_dataset = Dataset.from_dict({
"diff1": diff1_list,
"diff2": diff2_list,
"label": label_list
})
```
### Steps to reproduce the bug
1. Initialize a large 3-tuple, e.g. 26,000,000
2. Use Dataset.from_dict() to load
### Expected behavior
Dataset.from_dict() run successfully
### Environment info
sentence-transformers 3.3.1
Contributor guide
Assessment
This issue has not been assessed yet.