Calling `tresnet = resnet.layers |> torch` multiple times will increase GPU Memory-Usage

For example this code will increase memory usage of GPU.

```julia
for i in 1:100 tresnet = resnet.layers |> torch end
```

And this code will occur same phenomena

```julia
using BenchmarkTools
@btime tresnet(tip)
```

Here is my environment:

Ubuntu 16.04 with 2x1080Ti

```
(@v1.4) pkg> st
Status `~/.julia/environments/v1.4/Project.toml`
[587475ba] Flux v0.10.4
[dbeba491] Metalhead v0.5.0
[6a2ea274] Torch v0.1.1
```

```
Thu Jul 2 02:17:06 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:03:00.0 Off | N/A |
| 0% 51C P5 13W / 280W | 0MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 00000000:04:00.0 Off | N/A |
| 0% 52C P5 17W / 280W | 0MiB / 11177MiB | 3% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

```

Reproduce the repeated `resnet.layers |> torch` conversion and the `@btime tresnet(tip)` case under the reported Julia, Flux, Metalhead, and Torch versions. Trace the conversion and invocation entry points to identify what retains GPU memory; done means repeated conversions or benchmarks no longer cause unbounded GPU memory growth.

Written by the indexing model from the issue text.