pytorch / pytorch/ignite

Allow tuple in param_scheduler

Open
#438 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement question
Dominant language
Python
Stars
4.8k
Forks
726
Avg merge
5d 21h
Merged PRs (30d)
5

Description

Context:
I'd like to follow Leslie paper and change the momentum in adam at the same time I am changing the lr.
Issues:
The momentum in adam is equivalent of beta 1 which is the first values of the tuples in the torch initializer:
Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False) (ie. this is 0.9). However, right now there is no way to use param_scheduler with tuples let alone specifying only one tuple.
Proposed solution:
Allow param_name to be a bit more complex where you can specify things such as betas[0] (or something in the line of what pydash is doing pydash.get(param_gp[0],"betas[0]") which would require to basically use base_get) that will extract the first tuple from the betas.
I think it might be able to be done in the __call__ method in ParamScheduler

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in ParamScheduler.call and inspect how param_name currently resolves optimizer parameters. Define the supported tuple-path behavior described in the issue, including betas[0] and changing it alongside the learning rate, then verify that scheduling one tuple element leaves the other values unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.