Allow tuple in param_scheduler
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 726
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 5
Description
Context:
I'd like to follow Leslie paper and change the momentum in adam at the same time I am changing the lr.
Issues:
The momentum in adam is equivalent of beta 1 which is the first values of the tuples in the torch initializer:
Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False) (ie. this is 0.9). However, right now there is no way to use param_scheduler with tuples let alone specifying only one tuple.
Proposed solution:
Allow param_name to be a bit more complex where you can specify things such as betas[0] (or something in the line of what pydash is doing pydash.get(param_gp[0],"betas[0]") which would require to basically use base_get) that will extract the first tuple from the betas.
I think it might be able to be done in the __call__ method in ParamScheduler
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in ParamScheduler.call and inspect how param_name currently resolves optimizer parameters. Define the supported tuple-path behavior described in the issue, including betas[0] and changing it alongside the learning rate, then verify that scheduling one tuple element leaves the other values unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100