huggingface / huggingface/alignment-handbook
Cross Contamination in SFT Trainer
Open
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 490
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
Dear HuggingFace
I've noted that in run_cpt.py and run_sft.py, we introduce `packing=True`. However, we didn't provide `DataCollatorForCompletionOnlyLM` into SFTtrainer; would it introduce cross contamination in training?
referenece article: [Improving Hugging Face Training Efficiency Through Packing with Flash Attention](https://huggingface.co/blog/packing-with-FA2)
trl issue on github: https://github.com/huggingface/trl/issues/805
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.