huggingface / huggingface/dataset-viewer

Croissant for lmms-lab/LMMs-Eval-Lite is incorrect

Open
#3,145 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
899
Forks
130
Avg merge
9h 26m
Merged PRs (30d)
14

Description

The [croissant](https://huggingface.co/api/datasets/lmms-lab/LMMs-Eval-Lite/croissant) for the [lmms-lab/LMMs-Eval-Lite dataset](https://huggingface.co/datasets/lmms-lab/LMMs-Eval-Lite) is not correct.

And more specifically in the "gqa/semantic/dependencies" field:
```
{
"@type": "cr:Field",
"@id": "gqa/semantic/dependencies",
"name": "gqa/semantic/dependencies",
"description": "Column 'semantic' from the Hugging Face parquet file.",
"dataType": "sc:Integer",
"source": {
"fileSet": {
"@id": "parquet-files-for-config-gqa"
},
"extract": {
"column": "semantic"
}
// <==== Here the transform is missing!
}
```

I don't know why `"transform": { "jsonPath": "dependencies" }` is missing here, given that it has been correctly added to all other subfields.

For comparison, the croissant on mlcroissant repo for the same dataset, with the correct `gqa/semantic/dependencies` subfield: https://github.com/mlcommons/croissant/blob/0b5cdfdcea72025fa4f7eaf8384e92eda291a118/datasets/1.0/huggingface-lmms-eval-lite/metadata.json

Can be loaded both with and without beam without any issue.

Contributor guide

Open the contributing guide

Research direction

Compare the generated Croissant for lmms-lab/LMMs-Eval-Lite with the reference metadata in the linked mlcommons/croissant file, focusing on gqa/semantic/dependencies. Trace the dataset-viewer entry point that generates this field and verify that the resulting Croissant loads with and without Beam and includes the expected transform.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.