huggingface / huggingface/datasets

load_dataset does not load all of the data in my input file

Open
#6,432 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

I have 127 elements in my input dataset. When I do a len on the dataset after loaded, it is only 124 elements.

### Steps to reproduce the bug

train_dataset = nlp.load_dataset(data_args.dataset_path, name=data_args.qg_format, split=nlp.Split.TRAIN)
valid_dataset = nlp.load_dataset(data_args.dataset_path, name=data_args.qg_format, split=nlp.Split.VALIDATION)
logger.info(len(train_dataset))
logger.info(len(valid_dataset))

Both train and valid input are 127 items. However, they both only load 124 items. The input format is in json. At the end of the day, I am trying to create .pt files.

### Expected behavior

I see all 127 elements in my dataset when performing len

### Environment info

Python 3.10. CentOS operating system. nlp==0.40, datasets==2.14.5, transformers==4.26.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.