allenai / allenai/catwalk

Add a way to train a model before evaluating it

Offen
#4 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
154
Forks
18
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Motivation: Full fine-tuning is a baseline, or rather an upper bound, in many zero-shot and few-shot experiments. @pdasigi has explicitly asked for this.

As part of this work, we'll add a new Tango step to Catwalk that trains a model on a given task/dataset, or on multiple tasks/datasets at the same time. It should call into Tango's training functions to do so. We'll also need to add a method or two to Catwalk's `Model` class to make this happen. Then we'll do a full evaluation on all reasonable tasks and all reasonable models, to establish good baselines across the board. This might make for a good blog post, too.

As a stretch goal, we should also try to train adaptation methods like prompt tuning, prefix tuning, or even [IA3](https://arxiv.org/abs/2205.05638). There are some very nice implementations of some methods at https://github.com/r-three/t-few/tree/master/src.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.