Attributes to add/modify
- Dominant language
- JavaScript
- Stars
- 204
- Forks
- 40
- Avg merge
- 14m
- Merged PRs (30d)
- 15
Description
- [x] `Partial` Attribute for datasets that might not include all attributes. For example, some social media datasets require re-clawring.
- [x] `Unit`Add conversations in for tha attribute.
- [x] `Source` the source of the non-annotated data. `['web pages', 'public datasets', 'books', 'blogs', 'scientific articles', 'wikipedia', 'TV channels', 'LLM']`
- [x] `Test Split` rename to `Has_Splits`
- [x] `Domain` is for the domain of the dataset i.e., `['Law', 'Science', 'News', 'General', 'Quran', 'Culture', 'General']`
- [x] `Collection Style` should be either `['crawling', 'manual curation']`
- [x] `Annotation Style` to better represent how the dataset was annotated, and add another option for no annotation, for example, public datasets are inheritly annotated annotated `['Machine', 'Human', 'Inehrited']`
- [x] Define a list of venues for easier annotation
- [x] Rename Subsets to Dialect Subsets for better representation
- [ ] `Preprocessing` rather than `Tokenized` this general attributes define the preprocessning steps for the dataset i.e. `['tokenized','normalized','remove diacritization', 'other']`
- [x] Benchmarks are ill-defined in our definition; also, some datasets might include subsets other than dialects
Contributor guide
Research direction
Start by reviewing the repository's current dataset attribute definitions and the unchecked Preprocessing item. Confirm how Tokenized is represented, then determine the intended treatment of preprocessing steps, benchmarks, and non-dialect subsets; the work is done when these attribute definitions are consistently updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100