rope_factor args will be ignored if do rope_scaling during the initialization of McaGPTModel
Open
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 312
- Avg merge
- 1h 2m
- Merged PRs (30d)
- 2
Description
Main diff with mcore. cc @liu-zichen
https://github.com/alibaba/ROLL/blob/e1695f2dce9531f982196412a3ea3fd71d393ef1/mcore_adapter/src/mcore_adapter/models/model_factory.py#L274-L295
mcore: https://github.com/NVIDIA/Megatron-LM/blob/ea651a3b78b9b0c1dfd83975ff28853e38e5abaa/megatron/core/models/gpt/gpt_model.py#L75-L97
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with mcore_adapter/src/mcore_adapter/models/model_factory.py at lines 274-295 and compare it with the linked Megatron-LM gpt_model.py section. Reproduce or trace McaGPTModel initialization with rope_scaling, then verify that the rope_factor arguments are still honored when initialization performs rope scaling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100