karpathy / karpathy/pytorch-normalizing-flows
MADE bias note
- Dominant language
- Jupyter Notebook
- Stars
- 919
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Not a bug as much as an important note for MADE which are producing deltas on the input:
For the output node where all of its input connections masked, the bias for this neuron should also be masked and currently is not. The code inside MaskedLinear should be changed as follows:
```
def set_mask(self, mask): # called when the masks are created. passes in this mask.
mask = torch.from_numpy(mask.astype(np.uint8).T)
self.mask.data.copy_(mask)
# if all of the inputs are zero, need to ensure the bias
# is zeroed out!
self.bias_all_zero_mask = (mask.sum(dim=1)!=0).float()
def forward(self, input):
return F.linear(input, self.mask * self.weight, self.bias_all_zero_mask * self.bias)```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.