OptimalScale / OptimalScale/LMFlow
Some question on code of LISA
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 822
- PR merge metrics
- No merged PRs in 30d
Description
I have known that LISA's core code in src\lmflow\pipeline\finetuner.py, mainly in class DynamicLayerActivationCallback. I read it with Algorithm 1 Layerwise Importance Sampling AdamW (LISA) in paper aside.
So where is step2: Freeze all layers except the embedding and language modeling head layer? I can only find def freeze_all_layers(self) in class DynamicLayerActivationCallback, not excluding embedding and head layer
And i'm curious on the notation k in paper Algorithm 1:
step 4: Run AdamW for K iterations with ${η_t}_{t=ik}^{ik=k-1}$ Is the k same as K ?
My english is bad so tell me if any understanding problem,thanks for answering
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read src\lmflow\pipeline\finetuner.py, focusing on DynamicLayerActivationCallback and freeze_all_layers, then compare its behavior with Algorithm 1 in the LISA paper. Trace how layer freezing and optimizer iterations are represented, and document whether the implementation matches the paper or needs clarification; there is no test named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100