KFAC support in BatchNorm (eval mode)
- Dominant language
- Python
- Stars
- 617
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
Thanks for the repo! This is really a nice work.
I am planning to calculate the KFAC with backpack. But it raises the following error:
```
NotImplementedError: Extension saving to kfac does not have an extension for Module
```
My network is as follows:
```
model = nn.Sequential(
nn.Conv2d(1, 8, 3, stride=3),
nn.BatchNorm2d(8),
nn.ReLU(),
nn.Conv2d(8, 4, 3, stride=3),
nn.BatchNorm2d(4),
nn.ReLU(),
nn.Flatten(),
nn.Linear(36, 10))
loss = nn.CrossEntropyLoss()
```
When calculating the KFAC with:
```
model_ = extend(model.eval())
logits = model_(X)
loss = extend(loss_func)(logits, Y)
with backpack(KFAC(mc_samples=1000)):
loss.backward()
```
It raises the not implemented error. I am wondering whether calculating KFAC in a network with BN layers in the middle is supported by backpack? It seems like it should be supported, since it successfully works in ResNet.
Thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with the shown Sequential model using model.eval(), extend(), and backpack(KFAC(mc_samples=1000)) around loss.backward(). Start by tracing KFAC's extension handling for torch.nn.modules.batchnorm.BatchNorm2d; done means the same eval-mode network completes the backward pass without the NotImplementedError and produces KFAC quantities.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100