Issues with large Datasets

### Describe the bug

If the coco annotation file is too large the dataset will not be able to load it, not entirely sure were the issue is but I am guessing it is due to the code trying to load it all as one line into a dataframe. This was for object detections.

My current work around is the following code but would be nice to be able to do it without worrying about it also probably there is a better way of doing it:
`
dataset_dict = json.load(open("./local_data/annotations/train.json"))
df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'], 'categories': dataset_dict['categories']}, ignore_index=True)
train=Dataset.from_pandas(df)

dataset_dict = json.load(open("./local_data/annotations/validation.json"))
df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
'categories': dataset_dict['categories']}, ignore_index=True)
val = Dataset.from_pandas(df)
dataset_dict = json.load(open("./local_data/annotations/test.json"))
df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
'categories': dataset_dict['categories']}, ignore_index=True)
test = Dataset.from_pandas(df)
dataset = DatasetDict({'train': train, 'validation': val, 'test': test})
`

### Steps to reproduce the bug

1) step up directory in and have the json files in coco format

-local_data
|-images
|---1.jpg
|---2.jpg
|---....
|---n.jpg
|-annotations
|---test.json
|---train.json
|---validation.json

2) try to load local_data into a dataset if the file is larger than about 300kb it will cause an error.

### Expected behavior

That it loads the jsons preferably in the same format as it has done with a smaller size.

### Environment info

- `datasets` version: 3.3.3.dev0
- Platform: Linux-6.11.0-17-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.29.0
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0