clab / clab/dynet

restructuring weight decay?

Open
#1,244 0 comments 1 reaction 2 assignees Assigned to @redpony View on GitHub
enhancement
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

I find the way we currently handle weight-decay / regularization to be:

1. very badly documented
2. confusing to work with

In particular, there are currently two places to introduce weight-decay:
- Through a commandline option
- Through `set_weight_decay_lambda` in a `ParameterCollection`

The second option does not seem to work as intended (see #1243 ),
and the first and second options go through separate code paths.
Additionally, one cannot assign different weight decay values to individual parameters.

I suggest to move to the following behavior:

- Remove the commandline weight-decay option.
- `ParameterCollection` gets a weight-decay-lambda value on time of creation.
- for top-level `ParameterCollection`, the default is 0, and can be specified in the constructor.
- for sub-collections, the default is inherited from parent, but can be overriden in the constructor.
- Each `Parameters`/`LookupParameters` also gets its own weight-decay-lambda value.
- inherited from its parent.
- and can be set manually through a `.set_weight_decay_lambda` method.
- The value the optimizers actually look at is on the `Parameters`/`LookupParameters`.

The above proposal removes `set_weight_decay_lambda` from `ParameterCollection`, but I am also OK with keeping it for backward-compatibility.

If we want to keep the commandline option, I suggest it will be implemented as affecting the default value of the top-level `ParameterCollection`, only.

Thoughts?

I can give a shot at implementing it, if there is agreement.

---

PR #1242 now also fixes #1243 , by implementing part of the above proposal:

- Each collection / sub-collections inherits the weight-decay-lambda from its parent.
- The default weight-decay-lambda for a new collection is based on the commandline.
- Can use `set_weight_decay_lambda` on individual collections to change the default.

Changes from the above:
- No individual lambdas at the Parameters/LookupParemeters level.
- ...and trainers work as before
- No setting of lambda via constructors in Python.
- No setting of lambda in add_subcollection()
- Didn't remove the set_weight_decay_lambda method.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.