huggingface / huggingface/datasets
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
## Describe the bug
A clear and concise description of what the bug is.
using latest datasets=datasets-1.16.1-py3-none-any.whl
process my own multilingual dataset by following codes, and the number of rows in all dataset is 306000, the max_length of each sentence is 256:

then I get this error:

I have seen the issue in #2134 and #2150, so I don't understand why latest repo still can't deal with big dataset.
## Environment info
- `datasets` version:
- Platform: linux docker
- Python version: 3.6
Contributor guide
Assessment
This issue has not been assessed yet.