huggingface / huggingface/datasets

Specify dataset columns types in typehint

Open
#7,719 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Feature request

Make dataset optionaly generic to datasets usage with type annotations like it was done in `torch.Dataloader` https://github.com/pytorch/pytorch/blob/134179474539648ba7dee1317959529fbd0e7f89/torch/utils/data/dataloader.py#L131

### Motivation

In MTEB we're using a lot of datasets objects, but they're a bit poor in typehints. E.g. we can specify this for dataloder

```python
from typing import TypedDict
from torch.utils.data import DataLoader

class CorpusInput(TypedDict):
title: list[str]
body: list[str]

class QueryInput(TypedDict):
query: list[str]
instruction: list[str]

def queries_loader() -> DataLoader[QueryInput]:
...

def corpus_loader() -> DataLoader[CorpusInput]:
...
```
But for datasets we can only specify columns in type in comments
```python
from datasets import Dataset

QueryDataset = Dataset
"""Query dataset should have `query` and `instructions` columns as `str` """
```

### Your contribution

I can create draft implementation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.