allenai / allenai/natural-instructions

Human evaluation of the tasks

Ouverte
#206 3 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
1k
Forks
197
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

# Purpose
We need to ensure the quality of the presented tasks.
One way to do this is to ask human annotators (crowd workers) to read our instructions and answer them.

# Stages
I am assuming that we're gonna use AMTI for this: https://github.com/allenai/amti
- [x] Have a crowdsourcing template with place-holders for the instructions / positive examples / negative examples.
- [x] Have a script to read take the task name as a parameter and spit out a subset of it in an appropriate format. The resulting file should have all the placeholders needed for the template file.
- [ ] Evaluate a few of our tasks with this architecture to make sure that we have a reasonable pipeline.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.