google / google/flax

Bug in MinimalGatedUnit

Open
#4,608 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

https://github.com/google/flax/blob/d59132d9b17cdf2333ef72f6d7c96a21d81b71ba/flax/linen/recurrent.py#L725

This should be `1 - f`, according to the paper. Confusion arose around the effect of the "forget" gate (in LSTM and GRU papers, information is passed through when `f` is high, but in MGU paper it is the opposite). Variable `f` from the MGU paper, is effectively `1 - f` in Flax (it is the portion that is contributes to short-term response, or `n` in Flax-speak). From the paper:

> In MGU, the forget gate f_t is first generated, and the element-wise product between 1 - f_t and h_{t−1} becomes part of the new hidden state h_t. The portion of h_{t-1} that is "forgotten" (f_t h_{t−1}) is combined with x_t to produce h_bar_t, the short-term response. A portion of h_bar_t (determined again by f_t) form the second part of h_t.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.