Parameter `grad`s don't get initialized with `BatchL2Grad` and BatchNorm

BatchL2Grad, perhaps naturally, raises an error when it sees a BatchNorm, since batch normalization mixes gradients in a way that makes the individual contribution hard to discern.
The error says I can ignore it, if I know what I'm doing. I can't say I completely do, but if I ignore it, I do indeed get both `grad`s and `batch_l2`s on the top levels of my mode, which aren't using batch-norm.
I'm happy with that.

My problem is that the lower level parameters - which do use batch norm - don't just have a None `batch_l2`, but also a None `grad`.
So my model doesn't train at all.
This seems wrong, since `grad` is indeed computable, as witnessed by PyTorch being able to do so fine without backpack.

Is there a way I can get `batch_l2`s on as many of my parameters as possible, but `grad`s on everything?

I an do this now by first calling `backward()` without backpack, and then calling it again inside `with backpack(BatchL2Grad()):`, but that seems wasteful.