f-dangel / f-dangel/backpack

KFAC support in BatchNorm (eval mode)

Open
#259 16 comments 1 reaction 0 assignees View on GitHub
good first issue
Dominant language
Python
Stars
617
Forks
57
PR merge metrics
No merged PRs in 30d

Description

Hi,

Thanks for the repo! This is really a nice work.
I am planning to calculate the KFAC with backpack. But it raises the following error:

```
NotImplementedError: Extension saving to kfac does not have an extension for Module
```

My network is as follows:
```
model = nn.Sequential(
nn.Conv2d(1, 8, 3, stride=3),
nn.BatchNorm2d(8),
nn.ReLU(),
nn.Conv2d(8, 4, 3, stride=3),
nn.BatchNorm2d(4),
nn.ReLU(),
nn.Flatten(),
nn.Linear(36, 10))
loss = nn.CrossEntropyLoss()
```

When calculating the KFAC with:
```
model_ = extend(model.eval())
logits = model_(X)
loss = extend(loss_func)(logits, Y)
with backpack(KFAC(mc_samples=1000)):
loss.backward()
```

It raises the not implemented error. I am wondering whether calculating KFAC in a network with BN layers in the middle is supported by backpack? It seems like it should be supported, since it successfully works in ResNet.

Thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with the shown Sequential model using model.eval(), extend(), and backpack(KFAC(mc_samples=1000)) around loss.backward(). Start by tracing KFAC's extension handling for torch.nn.modules.batchnorm.BatchNorm2d; done means the same eval-mode network completes the backward pass without the NotImplementedError and produces KFAC quantities.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.