[Bug] AssertionError: neat_packing + context_parallel alignment mismatch (sub-sequence length not aligned to 2 * cp_size)
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 312
- Avg merge
- 1h 2m
- Merged PRs (30d)
- 2
Description
When training with Megatron Trainer’s backend in LLaMA-Factory, enabling both neat_packing and context_parallel_size (CP) results in an AssertionError: neat_packing + context_parallel alignment mismatch (sub-sequence length not aligned to 2 * cp_size)
config:
........
cut_off_len: 32768
sequence_packin: true
context_parallel_size: 4
.........
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the Megatron Trainer backend path that handles neat_packing with context_parallel_size; reproduce the failure using cut_off_len 32768, sequence packing, and context_parallel_size 4. Trace the reported alignment assertion and establish the intended behavior for sub-sequence lengths; done means the configuration no longer raises the mismatch and a regression test covers it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100