huggingface / huggingface/datasets
load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
When loading some jsonl from redpajama-data-1T github source [togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) fails due to one row of the file containing an extra field called **symlink_target: string>**.
When deleting that line the loading is successful.
We also tried loading this file with the discrepancy using this function and it is successful
```python
os.environ["RED_PAJAMA_DATA_DIR"] ="/path_to_local_copy_of_RedPajama-Data-1T"
ds = load_dataset('togethercomputer/RedPajama-Data-1T', 'github',cache_dir="/path_to_folder_with_jsonl",streaming=True)['train']
```
### Steps to reproduce the bug
Steps to reproduce the behavior:
1. Load one jsonl from the redpajama-data-1T
```bash
wget https://data.together.xyz/redpajama-data-1T/v1.0.0/github/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```
2.Load dataset will give error:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```
_TypeError: Couldn't cast array of type
Struct
>, **symlink_target: string>**
to
{'content_hash': Value(dtype='string', id=None),
'timestamp': Value(dtype='string', id=None),
'source': Value(dtype='string', id=None),
'line_count': Value(dtype='int64', id=None),
'max_line_length': Value(dtype='int64', id=None),
'avg_line_length': Value(dtype='float64', id=None),
'alnum_prop': Value(dtype='float64', id=None),
'repo_name': Value(dtype='string', id=None),
'id': Value(dtype='string', id=None),
'size': Value(dtype='string', id=None),
'binary': Value(dtype='bool', id=None),
'copies': Value(dtype='string', id=None),
'ref': Value(dtype='string', id=None),
'path': Value(dtype='string', id=None),
'mode': Value(dtype='string', id=None),
'license': Value(dtype='string', id=None),
'language': [{'name': Value(dtype='string', id=None), 'bytes': Value(dtype='string', id=None)}]}_
3. To remove the line causing the problem that includes the **symlink_target: string>** do:
```bash
sed -i '112252d' filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```
4. Rerun the loading function now is succesful:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```
### Expected behavior
Have a clean dataset without discrepancies on the jsonl fields or have the load_dataset('json',...) method not error out.
### Environment info
- `datasets` version: 2.14.1
- Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
Contributor guide
Assessment
This issue has not been assessed yet.