Croissant refers to incomplete parquet branch in native parquet datasets

The Croissant file exposed by HuggingFace seems to correspond to the parquet branch of the dataset, even when the dataset is native parquet:

- https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet

- https://huggingface.co/datasets/ai4bharat/sangraha

- https://huggingface.co/datasets/BleachNick/UltraEdit_500k

IIUC, the parquet branch is not complete for datasets >5GB (not exactly like that since the 5GB are per split), but overall the branch can be often incomplete for large datasets. There are exceptions though, in this dataset the Parquet branch seems complete:

- https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

Instead, there should be a way of retrieving a Croissant referring to the main native-parquet branch. Maybe for backward compatibility it would be better to expose both Croissant files (parquet branch and main branch) although exposing only the "complete" one could also be an option.

Start by comparing the Croissant output for the linked Hugging Face datasets with their native-parquet and parquet branches. Trace the existing Croissant generation entry point and determine how branch selection works; done means the output can represent the complete native-parquet data while the backward-compatibility choice is resolved and verified against the listed examples.

Written by the indexing model from the issue text.