huggingface / huggingface/datasets

struct.error: 'i' format requires -2147483648 <= number <= 2147483647

Open
#3,464 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

## Describe the bug
A clear and concise description of what the bug is.

using latest datasets=datasets-1.16.1-py3-none-any.whl
process my own multilingual dataset by following codes, and the number of rows in all dataset is 306000, the max_length of each sentence is 256:
![image](https://user-images.githubusercontent.com/30341159/146865779-3d25d011-1f42-4026-9e1b-76f6e1d172e9.png)

then I get this error:
![image](https://user-images.githubusercontent.com/30341159/146865844-e60a404c-5f3a-403c-b2f1-acd943b5cdb8.png)

I have seen the issue in #2134 and #2150, so I don't understand why latest repo still can't deal with big dataset.

## Environment info

- `datasets` version:
- Platform: linux docker
- Python version: 3.6

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.