Multiple independent models, only one requires apex.amp, crash in non-amp CPU model

I have a use-case where I have a "main" model which is trained with apex.amp at opt_level "O1", and all is fine. But I also have a small supplementary model which does not need mixed precision training and is trained on CPU. When apex.amp is enabled, training the second model (after the first model was trained) crashes with:

File "model.py"
  pred_logits = model(logits)
File "venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
  result = self.forward(*input, **kwargs)
File "model.py", in forward
  return self.linear(x)
File "venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
  result = self.forward(*input, **kwargs)
File "venv/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 87, in forward
  return F.linear(input, self.weight, self.bias)
File "venv/lib/python3.6/site-packages/apex/amp/wrap.py", line 28, in wrapper
  return orig_fn(*new_args, **kwargs)
File "venv/lib/python3.6/site-packages/torch/nn/functional.py", line 1370, in linear
  ret = torch.addmm(bias, input, weight.t())
File "venv/lib/python3.6/site-packages/apex/amp/wrap.py", line 21, in wrapper
  args[i] = utils.cached_cast(cast_fn, args[i], handle.cache)
File "venv/lib/python3.6/site-packages/apex/amp/utils.py", line 97, in cached_cast
  if cached_x.grad_fn.next_functions[1][0].variable is not x:
AttributeError: 'NoneType' object has no attribute 'next_functions'

This is happening with pytorch 1.3.1 and apex 2ca894da7be755711cbbdf56c74bb7904bfd8417 (latest master), and also happened with 82dac9c9419035110d1ccc49b2608681337903ed.

I'm not sure if this is a bug or me using apex.amp incorrectly - I see that the docs say that amp.initialize should be called only once (which is the case), but does this mean that all models to be used in the process must be passed? Is there a way around this? In this case the models are very unrelated and initializing them at once would be quite inconvenient.

I also created a simple repro - it crashes, but if we remove amp initialization or move the second model to GPU, the crash does not happen:

import torch
from apex import amp
from torchvision.models import resnet34
from torch.optim import SGD

device = torch.device('cuda')
model = resnet34()
optimizer = SGD(model.parameters(), lr=1e-2)
model.to(device)

use_amp = True
if use_amp:
    model, optimizer = amp.initialize(model, optimizer, opt_level='O1')
model(torch.randn(1, 3, 224, 224).to(device))

another_model = resnet34()
output = another_model(torch.randn(1, 3, 224, 224))
print(output.shape)

Start with the minimal reproduction in the issue, then inspect apex/amp/wrap.py and apex/amp/utils.py at the cached_cast call shown in the traceback. Reproduce the failure with PyTorch 1.3.1 and the cited Apex revisions; done means the AMP-initialized GPU model still runs while the independent CPU model no longer crashes.