huggingface / huggingface/trl

Add multi-GPU precompute test coverage for KTO

Open
#6,471 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

#6403 fixes precompute for DPO **and** KTO but only adds a DPO distributed test (`test_dpo_precompute_ref_log_probs`, ddp/zero2/zero3). Add the KTO mirror so the duplicated paths can't silently diverge: `trl/scripts/kto.py --precompute_ref_log_probs --eval_strategy epoch` (two-pass, the case that broke ZeRO-3), same configs. Check `kto.py` wires `eval_dataset` like `dpo.py`; exclude FSDP2 until Issue 1 lands.

Contributor guide

Open the contributing guide

Research direction

Start with the existing test_dpo_precompute_ref_log_probs distributed coverage and compare its ddp, zero2, and zero3 configurations. Then inspect trl/scripts/kto.py alongside dpo.py for eval_dataset wiring, and run the KTO equivalent with --precompute_ref_log_probs --eval_strategy epoch. Done means the KTO two-pass case is covered for the same configurations while FSDP2 remains excluded.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.