Previous line repeated xxx more times

After several epochs, the code raises some errors as below:

Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'",)
Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32768.0

(a few epochs later...)

File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/root/anaconda3/lib/python3.6/site-packages/apex/amp/wrap.py", line 28, in wrapper
return orig_fn(*new_args, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/apex/amp/wrap.py", line 28, in wrapper
return orig_fn(*new_args, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/apex/amp/wrap.py", line 28, in wrapper
return orig_fn(*new_args, **kwargs)
[Previous line repeated 28 more times]
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1369, in linear
ret = torch.addmm(bias, input, weight.t())
File "/root/anaconda3/lib/python3.6/site-packages/apex/amp/wrap.py", line 28, in wrapper
return orig_fn(*new_args, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/apex/amp/wrap.py", line 28, in wrapper
return orig_fn(*new_args, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/apex/amp/wrap.py", line 28, in wrapper
return orig_fn(*new_args, **kwargs)
[Previous line repeated 28 more times]
RuntimeError: CUDA out of memory. Tried to allocate 1.17 GiB (GPU 0; 10.76 GiB total capacity; 7.52 GiB already allocated; 591.56 MiB free; 1.74 GiB cached)

How can I solve it?

My environment:
cuda 10.0/10.1
pytorch1.2
python 3.6
GeForce RTX 2080ti

Start with the traceback in apex/amp/wrap.py and the reported PyTorch linear call, then review the mixed-precision setup and CUDA extension warning in the issue. Reproduce the failure with the stated CUDA, PyTorch, Python, and GPU versions; done means identifying a confirmed Apex cause and documenting a verified resolution.