LAION-AI / LAION-AI/Open-Assistant
Curate SFT-9 dataset mixes
Open
@echo0x22 is already working on this.
Since May 13, 2023.
data
ml
research
- Dominant language
- Python
- Stars
- 37.4k
- Forks
- 3.3k
- PR merge metrics
- No merged PRs in 30d
Description
Iterate on the SFT-8 dataset mixes to create pretraining and final SFT mixes for SFT-9. This requires investigating the quality and usefulness of the datasets. Community input welcome below. See the sft8_training branch for the code state corresponding to the below SFT-8 configs.
SFT-8 pretraining mix
datasets:
- gpteacher_roleplay:
val_split: 0.05
- red_pajama:
fraction: 0.25
max_val_set: 1000
- wizardlm_70k:
val_split: 0.05
max_val_set: 500
- joke:
val_split: 0.05
- poem_instructions:
val_split: 0.025
- oa_stackexchange:
val_split: 0.05
fraction: 0.1
max_val_set: 1000
- tell_a_joke:
val_split: 0.05
max_val_set: 250
- webgpt:
val_split: 0.05
max_val_set: 250
- gpt4all:
val_split: 0.01
max_val_set: 1000
- alpaca_gpt4:
val_split: 0.025
max_val_set: 250
- code_alpaca:
val_split: 0.05
max_val_set: 250
- vicuna:
max_val_set: 250
- oig_file:
source_url: https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl
max_count: 10000
min_length: 250
val_split: 0.05
max_val_set: 250
- minimath:
val_split: 0.05
- humaneval_mbpp_codegen_qa:
val_split: 0.05
- humaneval_mbpp_testgen_qa:
val_split: 0.05
- grade_school_math_instructions:
val_split: 0.05
- recipes:
val_split: 0.05
- cmu_wiki_qa:
val_split: 0.05
- oa_wiki_qa_bart_10000row:
val_split: 0.05
max_val_set: 250
- prosocial_dialogue:
fraction: 0.1
max_val_set: 250
- explain_prosocial:
fraction: 0.075
max_val_set: 250
- soda:
fraction: 0.25
max_val_set: 1000
- oa_leet10k:
val_split: 0.05
max_val_set: 250
- dolly15k:
val_split: 0.05
max_val_set: 300
SFT-8 final SFT mix
datasets:
- oasst_export:
lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk"
input_file_path: 2023-05-06_OASST_labels.jsonl.gz
val_split: 0.05
- vicuna:
val_split: 0.05
max_val_set: 800
fraction: 0.4
- dolly15k:
val_split: 0.05
max_val_set: 300
- grade_school_math_instructions:
val_split: 0.05
- code_alpaca:
val_split: 0.05
max_val_set: 250
- red_pajama:
fraction: 0.05
max_val_set: 1000
- wizardlm_70k:
val_split: 0.05
max_val_set: 500
fraction: 0.4
- poem_instructions:
fraction: 0.5
val_split: 0.025
Leading on this: @0x22almostEvil
Some initial requests from community include removal or reduction/filtering of prosocial_dialogue and explain_prosocial datasets from pretraining.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.