Unify dataset_name and dataset_cls
Open
data module
enhancement
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
Currently we are using `dataset_name` in data config, but it is not intuitive since that’s sometimes a dataset class.
1. Remove `dataset_name`, always use `dataset_cls` to make it easy to understand.
2. Support `dataset_cls` to load from path to support a custom `dataset_cls`.
3. Keep the compatibility of `dataset_name` for now, and remove all of them later.
```python
# [Before]
# for supported dataset
dataset_name: OpenMathInstruct-2
# for custom dataset
dataset_name: ResponseDataset
train_data_path:
# [After]
# for supported dataset
dataset_cls: OpenMathInstruct2Dataset
# for custom dataset
dataset_cls: ResponseDataset
train_data_path:
```
Contributor guide
Assessment
This issue has not been assessed yet.