huggingface / huggingface/datasets
Specify dataset columns types in typehint
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
Make dataset optionaly generic to datasets usage with type annotations like it was done in `torch.Dataloader` https://github.com/pytorch/pytorch/blob/134179474539648ba7dee1317959529fbd0e7f89/torch/utils/data/dataloader.py#L131
### Motivation
In MTEB we're using a lot of datasets objects, but they're a bit poor in typehints. E.g. we can specify this for dataloder
```python
from typing import TypedDict
from torch.utils.data import DataLoader
class CorpusInput(TypedDict):
title: list[str]
body: list[str]
class QueryInput(TypedDict):
query: list[str]
instruction: list[str]
def queries_loader() -> DataLoader[QueryInput]:
...
def corpus_loader() -> DataLoader[CorpusInput]:
...
```
But for datasets we can only specify columns in type in comments
```python
from datasets import Dataset
QueryDataset = Dataset
"""Query dataset should have `query` and `instructions` columns as `str` """
```
### Your contribution
I can create draft implementation
Contributor guide
Assessment
This issue has not been assessed yet.