bigscience-workshop / bigscience-workshop/evaluation

Wrap evaluation benchmark using HF-trainer

Open
#61 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
42
Forks
24
PR merge metrics
No merged PRs in 30d

Description

This might sounds like a bit of re-structuring but for the sake of future compatibility, I propose the following,
1. Move to `huggingface` trainer: This will help the repo to automatically adapt to `deepspeed` and all the exclusive features of transformers library.
2. We don't have to re-invent the wheel. Given that we are using huggingface trainer, we only need to implement the following functions for a trainer for different tasks.
-- `data_loader`
-- `DataCollator`
-- `compute_metrics`
-- `predictions` (if needed)
3. In case if we want to `finetune` our full model, we don't have to change a lot in the surface level.

I would love to take some responsibility if needed. Let me know. @jaketae @tianjianjiang @wilsonyhlee

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.