linkedin / linkedin/Liger-Kernel
[QnA]: Why `cos` and `sin` is expected to be `hdim`, not `hdim//2`?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 603
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 47
Description
I have a question regarding to the qwen2_vl MRope. From my understanding is as follows:
```
full_cos = torch.cat([cos_halfdim, cos_halfdim], dim=-1)
full_sin = torch.cat([sin_halfdim, sin_halfdim], dim=-1)
```
However from the unit tests and the code here,
https://github.com/linkedin/Liger-Kernel/blob/2845fe8363a6f40a265ec8102523fe4c0ded068e/src/liger_kernel/ops/qwen2vl_mrope.py#L7
the `cos` and `sin` are full_cos and full_sin instead. Is there a reason to not just pass half of the cos and sin to save memory movement?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/liger_kernel/ops/qwen2vl_mrope.py at the linked lines, then inspect the related unit tests and Qwen2-VL MRoPE behavior. Determine whether the kernel intentionally requires full-dimension cos and sin inputs or can accept half-dimension values, and document the reasoning or expected interface when the question is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100