bigscience-workshop / bigscience-workshop/evaluation
Wrap evaluation benchmark using HF-trainer
- Dominant language
- Python
- Stars
- 42
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Description
This might sounds like a bit of re-structuring but for the sake of future compatibility, I propose the following,
1. Move to `huggingface` trainer: This will help the repo to automatically adapt to `deepspeed` and all the exclusive features of transformers library.
2. We don't have to re-invent the wheel. Given that we are using huggingface trainer, we only need to implement the following functions for a trainer for different tasks.
-- `data_loader`
-- `DataCollator`
-- `compute_metrics`
-- `predictions` (if needed)
3. In case if we want to `finetune` our full model, we don't have to change a lot in the surface level.
I would love to take some responsibility if needed. Let me know. @jaketae @tianjianjiang @wilsonyhlee
Contributor guide
Assessment
This issue has not been assessed yet.