[Question] motivation and recommended usage of --allgather-cp

Your Question

Hi, thanks for the project.

I’m trying to understand the motivation behind --allgather-cp, especially because I noticed that:

• it appears to have been introduced together with GLM5 support
• in the Megatron backend, it changes the CP data layout quite significantly

From reading the code, my current understanding is:

• without --allgather-cp, CP uses the more traditional zigzag-style per-sample slicing
• with --allgather-cp, the batch is first globally concatenated, then split into contiguous chunks across CP ranks
• after forward, the response-aligned outputs are redistributed back into the zigzag-style view expected by downstream loss logic

What I’m still confused about is the tradeoff.

My intuition was that zigzag CP is useful because it balances computation for each sequence more evenly across ranks.
So I’m trying to understand why the “global concat + contiguous split” layout is preferable here.

Thanks.

What I've Tried

The relevant places I looked at are:

• slime/backends/megatron_utils/data.py
• slime/backends/megatron_utils/loss.py
• slime_plugins/models/glm5/glm5.py

Environment (if relevant)

• slime version:
• Python version:
• PyTorch version:
• CUDA/ROCm version:
• GPU type and count:
• OS:

Additional Context

No response

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and FAQ and my question is not answered there.
• I have searched for existing issues and my question has not been asked before.

Read slime/backends/megatron_utils/data.py and loss.py alongside slime_plugins/models/glm5/glm5.py. Trace how --allgather-cp changes layout and output redistribution, then establish the rationale, tradeoffs, and recommended usage from the implementation. Done means providing a documented, evidence-based answer to the question.