huggingface / huggingface/datasets

load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields

Open
#6,162 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

When loading some jsonl from redpajama-data-1T github source [togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) fails due to one row of the file containing an extra field called **symlink_target: string>**.
When deleting that line the loading is successful.

We also tried loading this file with the discrepancy using this function and it is successful
```python
os.environ["RED_PAJAMA_DATA_DIR"] ="/path_to_local_copy_of_RedPajama-Data-1T"
ds = load_dataset('togethercomputer/RedPajama-Data-1T', 'github',cache_dir="/path_to_folder_with_jsonl",streaming=True)['train']
```

### Steps to reproduce the bug

Steps to reproduce the behavior:
1. Load one jsonl from the redpajama-data-1T
```bash
wget https://data.together.xyz/redpajama-data-1T/v1.0.0/github/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```

2.Load dataset will give error:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```

_TypeError: Couldn't cast array of type
Struct
>, **symlink_target: string>**
to
{'content_hash': Value(dtype='string', id=None),
'timestamp': Value(dtype='string', id=None),
'source': Value(dtype='string', id=None),
'line_count': Value(dtype='int64', id=None),
'max_line_length': Value(dtype='int64', id=None),
'avg_line_length': Value(dtype='float64', id=None),
'alnum_prop': Value(dtype='float64', id=None),
'repo_name': Value(dtype='string', id=None),
'id': Value(dtype='string', id=None),
'size': Value(dtype='string', id=None),
'binary': Value(dtype='bool', id=None),
'copies': Value(dtype='string', id=None),
'ref': Value(dtype='string', id=None),
'path': Value(dtype='string', id=None),
'mode': Value(dtype='string', id=None),
'license': Value(dtype='string', id=None),
'language': [{'name': Value(dtype='string', id=None), 'bytes': Value(dtype='string', id=None)}]}_

3. To remove the line causing the problem that includes the **symlink_target: string>** do:
```bash
sed -i '112252d' filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```
4. Rerun the loading function now is succesful:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```

### Expected behavior

Have a clean dataset without discrepancies on the jsonl fields or have the load_dataset('json',...) method not error out.

### Environment info

- `datasets` version: 2.14.1
- Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.