LAION-AI / LAION-AI/Open-Assistant
Enforce and test common interface for datasets in model_training
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 37.4k
- Forks
- 3.3k
- PR merge metrics
- No merged PRs in 30d
Description
Currently each training dataset has it's own class in which some transformation of the base dataset is done. Our models rely on specific formats of the data, but this is never enforced and things are broken/might be broken in the future. With proper design and testing we can verify that this might not be happening in the future and find bugs in the dataset transformations that exist currently.
One possible solution for this would be to write a DatasetMixin, that extracts a very small subset of the data (a few samples are enough) and and checks whether the output format corresponds to the expected format. I identify the following todos:
- find different required dataset formats for SFT, RM and RL
- write mixin class
- let current datasets inherit from Mixin class
- write tests that perform the check which is implemented by the mixin
- document the tested datasets
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the model_training dataset classes and inventory the required formats for SFT, RM, and RL. Define the small-sample interface check, apply it to the current datasets, and add tests for the check and existing transformations. Done means the current datasets conform and the tested datasets are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100