alibaba / alibaba/ROLL

[Bug] AssertionError: neat_packing + context_parallel alignment mismatch (sub-sequence length not aligned to 2 * cp_size)

Open
#325 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
312
Avg merge
1h 2m
Merged PRs (30d)
2

Description

When training with Megatron Trainer’s backend in LLaMA-Factory, enabling both neat_packing and context_parallel_size (CP) results in an AssertionError: neat_packing + context_parallel alignment mismatch (sub-sequence length not aligned to 2 * cp_size)

config:
........
cut_off_len: 32768
sequence_packin: true
context_parallel_size: 4
.........

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Megatron Trainer backend path that handles neat_packing with context_parallel_size; reproduce the failure using cut_off_len 32768, sequence packing, and context_parallel_size 4. Trace the reported alignment assertion and establish the intended behavior for sub-sequence lengths; done means the configuration no longer raises the mismatch and a regression test covers it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.