huggingface / huggingface/dataset-viewer
Modalities not detected for some datasets using the Webdatasets format
- Dominant language
- Python
- Stars
- 899
- Forks
- 130
- Avg merge
- 9h 26m
- Merged PRs (30d)
- 14
Description
I have found 2 examples of the modality detection code failing to recognize modalities in text and image datasets using the Webdataset format:
* https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-dalle3-high-quality-captions
* https://huggingface.co/datasets/CaptionEmporium/midjourney-niji-1m-llavanext
I'm not sure where in the modality detection code that things are failing: https://github.com/huggingface/dataset-viewer/blob/main/services/worker/src/worker/job_runners/dataset/modalities.py
Contributor guide
Research direction
Read services/worker/src/worker/job_runners/dataset/modalities.py first, then reproduce modality detection for the two linked Webdataset examples. Trace why their text and image modalities are missed and verify that both datasets are recognized correctly after the fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100