huggingface / huggingface/dataset-viewer

The parameters of an endpoint should not change the response format

Open
#1,665 0 comments 0 reactions 0 assignees View on GitHub
api bug P2
Dominant language
Python
Stars
899
Forks
130
Avg merge
9h 26m
Merged PRs (30d)
14

Description

The optional parameters should only change the response's content, not structure.

For example, the `length` parameter in /rows reduces the number of returned rows.

But for /parquet, for example, if we ask for the config level (https://datasets-server.huggingface.co/parquet?dataset=mnist), we get the list of features along with the list of files, while we don't have features when we only ask for the dataset level (https://datasets-server.huggingface.co/parquet?dataset=mnist&config=mnist). Also, for /info, the structure of `dataset_info` is not the same for dataset level and config level.

For /size, the fields' names and types change depending on whether the config parameter is passed or not. For example, https://datasets-server.huggingface.co/size?dataset=mnist gives `.size.configs`, while https://datasets-server.huggingface.co/size?dataset=mnist&config=mnist give `.size.config`.

Similarly, the `failed` and `pending` entries are weird. They only show for "aggregated" levels (i.e., dataset if the response is generated at config level, dataset, and config if the response is generated at split level). Currently:
- /splits, dataset level
- /parquet, dataset level
- /info, dataset level
- /size, dataset level
- /opt-in-out-urls, dataset and config levels

About "failed" and "pending", also note that their type is different depending on the endpoint. Just one example: "failed" in /splits return the error, while "failed" in /parquet return the parameters of the previous job.

Also: in parquet and info, instead of not setting "split", we set it to None (which gives `null` in JSON, instead of not having the field).

Contributor guide

Open the contributing guide

Research direction

Start by comparing the documented responses for /rows, /parquet, /info, /size, /splits, and /opt-in-out-urls at their different levels and parameter combinations. Define the consistent fields and types required for each endpoint, including failed, pending, and split, then verify that the responses no longer change structure when optional parameters are used.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.