LAION-AI / LAION-AI/Open-Assistant
Labeling: Standards & Guidelines
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 37.4k
- Forks
- 3.3k
- PR merge metrics
- No merged PRs in 30d
Description
Currently, not only the evaluations, but also the definitions of the terms are often very dependent on a subjective perception. It would be advantageous if there were universally valid definitions of these terms.
Ideally, this would be combined with some examples so that a general understanding of the criteria can be established. If this would be embedded on the Training GUI or at least interlinked, that would be super helpful for everyone and potentially increases the rating quality.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing labeling terms, evaluation criteria, and the Training GUI referenced in the issue. Define universally applicable terms with examples, then document or link the guidance in the Training GUI so labelers can apply consistent criteria.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100