restructuring weight decay?
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
I find the way we currently handle weight-decay / regularization to be:
1. very badly documented
2. confusing to work with
In particular, there are currently two places to introduce weight-decay:
- Through a commandline option
- Through `set_weight_decay_lambda` in a `ParameterCollection`
The second option does not seem to work as intended (see #1243 ),
and the first and second options go through separate code paths.
Additionally, one cannot assign different weight decay values to individual parameters.
I suggest to move to the following behavior:
- Remove the commandline weight-decay option.
- `ParameterCollection` gets a weight-decay-lambda value on time of creation.
- for top-level `ParameterCollection`, the default is 0, and can be specified in the constructor.
- for sub-collections, the default is inherited from parent, but can be overriden in the constructor.
- Each `Parameters`/`LookupParameters` also gets its own weight-decay-lambda value.
- inherited from its parent.
- and can be set manually through a `.set_weight_decay_lambda` method.
- The value the optimizers actually look at is on the `Parameters`/`LookupParameters`.
The above proposal removes `set_weight_decay_lambda` from `ParameterCollection`, but I am also OK with keeping it for backward-compatibility.
If we want to keep the commandline option, I suggest it will be implemented as affecting the default value of the top-level `ParameterCollection`, only.
Thoughts?
I can give a shot at implementing it, if there is agreement.
---
PR #1242 now also fixes #1243 , by implementing part of the above proposal:
- Each collection / sub-collections inherits the weight-decay-lambda from its parent.
- The default weight-decay-lambda for a new collection is based on the commandline.
- Can use `set_weight_decay_lambda` on individual collections to change the default.
Changes from the above:
- No individual lambdas at the Parameters/LookupParemeters level.
- ...and trainers work as before
- No setting of lambda via constructors in Python.
- No setting of lambda in add_subcollection()
- Didn't remove the set_weight_decay_lambda method.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.