karpathy / karpathy/pytorch-normalizing-flows

MADE bias note

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
919
Forks
102
PR merge metrics
No merged PRs in 30d

Description

Not a bug as much as an important note for MADE which are producing deltas on the input:

For the output node where all of its input connections masked, the bias for this neuron should also be masked and currently is not. The code inside MaskedLinear should be changed as follows:

```
def set_mask(self, mask): # called when the masks are created. passes in this mask.
mask = torch.from_numpy(mask.astype(np.uint8).T)
self.mask.data.copy_(mask)
# if all of the inputs are zero, need to ensure the bias
# is zeroed out!
self.bias_all_zero_mask = (mask.sum(dim=1)!=0).float()

def forward(self, input):
return F.linear(input, self.mask * self.weight, self.bias_all_zero_mask * self.bias)```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.