Add a way to train a model before evaluating it
- Vorherrschende Sprache
- Python
- Sterne
- 154
- Forks
- 18
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
Motivation: Full fine-tuning is a baseline, or rather an upper bound, in many zero-shot and few-shot experiments. @pdasigi has explicitly asked for this.
As part of this work, we'll add a new Tango step to Catwalk that trains a model on a given task/dataset, or on multiple tasks/datasets at the same time. It should call into Tango's training functions to do so. We'll also need to add a method or two to Catwalk's `Model` class to make this happen. Then we'll do a full evaluation on all reasonable tasks and all reasonable models, to establish good baselines across the board. This might make for a good blog post, too.
As a stretch goal, we should also try to train adaptation methods like prompt tuning, prefix tuning, or even [IA3](https://arxiv.org/abs/2205.05638). There are some very nice implementations of some methods at https://github.com/r-three/t-few/tree/master/src.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.