allenai / allenai/catwalk

Adding models/methods/datasets

Offen
#27 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
154
Forks
18
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Motivation: Various people have asked for various additions to Catwalk already. It's risky because nobody is using Catwalk yet. But we have several people who said they want to (Pradeep, Matt/Hamish, Iz?, Ludwig).

Here are the sub-projects in order of importance:

* [Promptsource](https://github.com/bigscience-workshop/promptsource): This is the most requested one. The task would be to add a promptsource instance format to as many tasks as possible, and then evaluate various models with that format.
* Make sure we have all the tasks in P3. This might be a no-op after the first item.
* Few-shot prompting (for in-context learning). Nobody has explicitly asked for this. I think nobody asks because it's obvious that catwalk would have this.
* [Crossfit](https://github.com/INK-USC/CrossFit): Pradeep wants to use Crossfit. Those tasks should be fairly easy to add.
* T-Few seems like a good baseline for a lot of our work, so it might become a good benchmark set for a while.
* [BigBench](https://github.com/google/BIG-bench): Pradeep asked about those too, but backed off from it later. Could be a nice addition, but is at the bottom of this list on purpose.
* Prompt format that uses the "[channel method](https://arxiv.org/abs/2108.04106)" for decoder-only models. Nobody has asked for it, but it came up in our reading group. I thought I could verify it with a quick experiment, but I could not.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.