huggingface / huggingface/datasets

One or several metadata.jsonl were found, but not in the same directory or in a parent directory

Open
#5,761 3 comments 1 reaction 1 assignee Claimed by @albertvillanova View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

An attempt to generate a dataset from a zip archive using imagefolder and metadata.jsonl does not lead to the expected result. Tried all possible locations of the json file: the file in the archive is ignored (generated dataset contains only images), the file next to the archive like [here](https://huggingface.co/docs/datasets/image_dataset#imagefolder) leads to an error:
```
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File ~\PycharmProjects\testproj\venv\lib\site-packages\datasets\builder.py:1610, in GeneratorBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id)
1609 _time = time.time()
-> 1610 for key, record in generator:
1611 if max_shard_size is not None and writer._num_bytes > max_shard_size:

File ~\PycharmProjects\testproj\venv\lib\site-packages\datasets\packaged_modules\folder_based_builder\folder_based_builder.py:370, in FolderBasedBuilder._generate_examples(self, files, metadata_files, split_name, add_metadata, add_labels)
369 else:
--> 370 raise ValueError(
371 f"One or several metadata.{metadata_ext} were found, but not in the same directory or in a parent directory of {downloaded_dir_file}."
372 )
373 if metadata_dir is not None and downloaded_metadata_file is not None:

ValueError: One or several metadata.jsonl were found, but not in the same directory or in a parent directory of C:\Users\User\.cache\huggingface\datasets\downloads\extracted\f7fb7de25fb28ae63089974524f2d271a39d83888bc456d04aa3b3d45f33e6a6\ff0745a0-a741-4d9e-b228-a93b851adf61.png.

The above exception was the direct cause of the following exception:

DatasetGenerationError Traceback (most recent call last)
Cell In[3], line 1
----> 1 dataset = load_dataset("imagefolder", data_dir=r'C:\Users\User\data')

File ~\PycharmProjects\testproj\venv\lib\site-packages\datasets\load.py:1791, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
1788 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
1790 # Download and prepare data
-> 1791 builder_instance.download_and_prepare(
1792 download_config=download_config,
1793 download_mode=download_mode,
1794 verification_mode=verification_mode,
1795 try_from_hf_gcs=try_from_hf_gcs,
1796 num_proc=num_proc,
1797 storage_options=storage_options,
1798 )
1800 # Build dataset for splits
1801 keep_in_memory = (
1802 keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
1803 )

File ~\PycharmProjects\testproj\venv\lib\site-packages\datasets\builder.py:891, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
889 if num_proc is not None:
890 prepare_split_kwargs["num_proc"] = num_proc
--> 891 self._download_and_prepare(
892 dl_manager=dl_manager,
893 verification_mode=verification_mode,
894 **prepare_split_kwargs,
895 **download_and_prepare_kwargs,
896 )
897 # Sync info
898 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~\PycharmProjects\testproj\venv\lib\site-packages\datasets\builder.py:1651, in GeneratorBasedBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_splits_kwargs)
1650 def _download_and_prepare(self, dl_manager, verification_mode, **prepare_splits_kwargs):
-> 1651 super()._download_and_prepare(
1652 dl_manager,
1653 verification_mode,
1654 check_duplicate_keys=verification_mode == VerificationMode.BASIC_CHECKS
1655 or verification_mode == VerificationMode.ALL_CHECKS,
1656 **prepare_splits_kwargs,
1657 )

File ~\PycharmProjects\testproj\venv\lib\site-packages\datasets\builder.py:986, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
982 split_dict.add(split_generator.split_info)
984 try:
985 # Prepare split will record examples associated to the split
--> 986 self._prepare_split(split_generator, **prepare_split_kwargs)
987 except OSError as e:
988 raise OSError(
989 "Cannot find data file. "
990 + (self.manual_download_instructions or "")
991 + "\nOriginal error:\n"
992 + str(e)
993 ) from None

File ~\PycharmProjects\testproj\venv\lib\site-packages\datasets\builder.py:1490, in GeneratorBasedBuilder._prepare_split(self, split_generator, check_duplicate_keys, file_format, num_proc, max_shard_size)
1488 gen_kwargs = split_generator.gen_kwargs
1489 job_id = 0
-> 1490 for job_id, done, content in self._prepare_split_single(
1491 gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
1492 ):
1493 if done:
1494 result = content

File ~\PycharmProjects\testproj\venv\lib\site-packages\datasets\builder.py:1646, in GeneratorBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id)
1644 if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
1645 e = e.__context__
-> 1646 raise DatasetGenerationError("An error occurred while generating the dataset") from e
1648 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

1. Organize directory structure like in the docs:
folder/metadata.jsonl
folder/train.zip

2. Run load_dataset("imagefolder", data_dir='folder/metadata.jsonl', split='train')

### Expected behavior

Dataset generated with all additional features from metadata.jsonl

### Environment info

- `datasets` version: 2.11.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.0
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.