Question in offload.py: Moving activation to CPU does NOT reduce GPU memory.

I use my `cuda_active_bytes` function to measure the GPU memory before and after the code line below. I find moving activation to CPU does NOT reduce GPU memory.
https://github.com/facebookresearch/fairscale/blob/9f347f373e32ee5cad11a40b70b8e28a74b5e2d4/fairscale/experimental/nn/offload.py#L524
```python
def cuda_active_bytes():
torch.cuda.synchronize()
stats = torch.cuda.memory_stats()
current_active_byte = stats['active_bytes.all.current']
return current_active_byte
```
So actually all the activations generated by forward is still in GPU memory? If so, I think the code line above is redundant.

Start at fairscale/experimental/nn/offload.py#L524 and reproduce the report with the provided cuda_active_bytes function, including torch.cuda.synchronize() and torch.cuda.memory_stats(). Read the surrounding activation-offload path and determine whether the observed active_bytes.all.current value reflects the expected behavior. Done means explaining the discrepancy and whether the line is redundant, with evidence from the reproduction.

Written by the indexing model from the issue text.