huggingface / huggingface/datasets

Workflow for Tabular data

Open
#4,456 8 comments 2 reactions 0 assignees View on GitHub
enhancement generic discussion
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

Tabular data are treated very differently than data for NLP, audio, vision, etc. and therefore the worflow for tabular data in `datasets` is not ideal.

For example for tabular data, it is common to use pandas/spark/dask to process the data, and then load the data into X and y (X is an array of features and y an array of labels), then train_test_split and finally feed the data to a machine learning model.

In `datasets` the workflow is different: we use load_dataset, then map, then train_test_split (if we only have a train split) and we end up with columnar dataset splits, not formatted as X and y.

Right now, it is already possible to convert a dataset from and to pandas, but there are still many things that could improve the workflow for tabular data:
- be able to load the data into X and y
- be able to load a dataset from the output of spark or dask (as far as I know it's usually csv or parquet files on S3/GCS/HDFS etc.)
- support "unsplit" datasets explicitly, instead of putting everything in "train" by default

cc @adrinjalali @merveenoyan feel free to complete/correct this :)

Feel free to also share ideas of APIs that would be super intuitive in your opinion !

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.