ARBML / ARBML/masader

Attributes to add/modify

Open
#663 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
204
Forks
40
Avg merge
14m
Merged PRs (30d)
15

Description

- [x] `Partial` Attribute for datasets that might not include all attributes. For example, some social media datasets require re-clawring.
- [x] `Unit`Add conversations in for tha attribute.
- [x] `Source` the source of the non-annotated data. `['web pages', 'public datasets', 'books', 'blogs', 'scientific articles', 'wikipedia', 'TV channels', 'LLM']`
- [x] `Test Split` rename to `Has_Splits`
- [x] `Domain` is for the domain of the dataset i.e., `['Law', 'Science', 'News', 'General', 'Quran', 'Culture', 'General']`
- [x] `Collection Style` should be either `['crawling', 'manual curation']`
- [x] `Annotation Style` to better represent how the dataset was annotated, and add another option for no annotation, for example, public datasets are inheritly annotated annotated `['Machine', 'Human', 'Inehrited']`
- [x] Define a list of venues for easier annotation
- [x] Rename Subsets to Dialect Subsets for better representation
- [ ] `Preprocessing` rather than `Tokenized` this general attributes define the preprocessning steps for the dataset i.e. `['tokenized','normalized','remove diacritization', 'other']`
- [x] Benchmarks are ill-defined in our definition; also, some datasets might include subsets other than dialects

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the repository's current dataset attribute definitions and the unchecked Preprocessing item. Confirm how Tokenized is represented, then determine the intended treatment of preprocessing steps, benchmarks, and non-dialect subsets; the work is done when these attribute definitions are consistently updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.