huggingface / huggingface/datasets
Allow manual configuration of Dataset Viewer for datasets not created with the `datasets` library
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
#### **Problem Description**
Currently, the Hugging Face Dataset Viewer automatically interprets dataset fields for datasets created with the `datasets` library. However, for datasets pushed directly via `git`, the Viewer:
- Defaults to generic columns like `label` with `null` values if no explicit mapping is provided.
- Does not allow dataset creators to configure field mappings or suppress default fields unless the dataset is recreated and pushed using the `datasets` library.
This creates a limitation for creators who:
- Use custom workflows to prepare datasets (e.g., manifest files with audio-transcription mappings).
- Push large datasets directly via `git` and cannot easily restructure them to conform to the `datasets` library format.
#### **Proposed Solution**
Introduce a feature that allows dataset creators to manually configure the Dataset Viewer behavior for datasets not created with the `datasets` library. This could be achieved by:
1. **Using the YAML Metadata in `README.md`:**
- Add support for defining the dataset's field mappings directly in the `README.md` YAML section.
- Example:
```yaml
viewer:
fields:
- name: "audio"
type: "audio_path" / "text"
source: "manifest['audio']"
- name: "bambara_transcription"
type: "text"
source: "manifest['bambara']"
- name: "french_translation"
type: "text"
source: "manifest['french']"
```
With manifest being a csv or json like format file in the repository so that the viewer understands that it should look for the values of each field in that file.
#### **Benefits**
- Improves flexibility for dataset creators who push datasets via `git`.
- Enhances dataset discoverability and usability on the Hugging Face Hub by allowing creators to present meaningful field mappings without restructuring their data.
- Reduces overhead for creators of large or complex datasets.
#### **Examples of Use Case**
- An audio dataset with transcriptions in multiple languages stored in a `manifest.json` file, where the user wants the Viewer to:
- Display the `audio` column and Explicitly map features that he defined such as `bambara_transcription` and `french_translation` from the manifest.
Contributor guide
Assessment
This issue has not been assessed yet.