LAION-AI / LAION-AI/Open-Assistant
Add korean dataset
Open
@CertifiedJoon is already working on this.
Since Jul 6, 2023.
data
- Dominant language
- Python
- Stars
- 37.4k
- Forks
- 3.3k
- PR merge metrics
- No merged PRs in 30d
Description
In order for open-assistant to work in Korean, we are working on adding a Korean dataset.
Creating a dataset from zero to end is quite difficult. To efficiently add a dataset, we will proceed as follows.
Current progress is as follows.
- Crawl the dataset
- A dataset that includes Korean, such as wikihow, and can be used in the form of "instruction-fulfillment"
- Machine translation
- Obtain the Korean dataset by machine-translating the en dataset from the OA Dataset List.
- Machine translation may not be perfect, but I think it can be filtering in the labeling task.
- In addition, if you have a list of datasets that I can refer to, please share!
These datasets are expected to be used for labeling and learning for RLHF.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.