deepspeedai / deepspeedai/DeepSpeed
[BUG] Is flash attention 3 compatible with Ulysses or not?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
Hello, I've been training Qwen3 recently with FA3 and Ulysses sp > 1, but I get nan logits after forward:
(WorkerDict pid=8874) WARN: rank 1 non-finite policy loss. pg_loss_finite=False ppo_kl_finite=False. old_log_prob_stats={'shape': (3, 4096), 'nan': 136, 'inf': 0, 'min': -26.433170318603516, 'max': 0.0} log_prob_stats={'shape': (3, 4096), 'nan': 0, 'inf': 0, 'min':
To Reproduce
Steps to reproduce the behavior:
- Using verl for dapo training.
- Using fa3 and Ulysses sp > 1.
- Observing that loss is nan.
- Print forward logits and find out that those are already nan.
Expected behavior
Loss should be not nan.
Screenshots
System info (please complete the following information):
- OS: [Ubuntu 18.04]
- GPU count and types [H200 * 16]
- Interconnects (if applicable) [two machines connected with 100 Gbps IB]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report provides no source files or tests to start from. Reproduce DAPO training with FA3 and Ulysses sequence parallelism greater than 1 on the stated Ubuntu and H200 setup, then trace the forward logits before the loss. Done means determining whether this combination is compatible and identifying why non-finite logits occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100