Thrice as much memory for AlexNet on Caltech256 in Julia than in Python. Why?

I am having a severe problem with training AlexNet (see [alexnet.jl](https://gist.github.com/hesseltuinhof/01d52e5ba64546bf6b806b3ffcc10c3f#file-alexnet-jl)) in Julia (0.5.2) on my GPU (12gb mem).

I am training on `Caltech256` dataset (see [main.jl](https://gist.github.com/hesseltuinhof/01d52e5ba64546bf6b806b3ffcc10c3f#file-main-jl))

Julia variant: I am out of memory when starting. See the following log:
```julia
julia> include("main.jl")
[15:36:03] src/io/iter_image_recordio_2.cc:135: ImageRecordIOParser2: ../../../data/caltech256-train.rec, use 4 threads for decoding..
[15:36:03] src/io/iter_image_recordio_2.cc:135: ImageRecordIOParser2: ../../../data/caltech256-val.rec, use 4 threads for decoding..
INFO: Start training on MXNet.mx.Context[GPU0]
INFO: Initializing parameters...
INFO: Creating KVStore...
[15:36:09] src/operator/././cudnn_algoreg-inl.h:65: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
INFO: TempSpace: Total 1345 MB allocated on GPU0
INFO: Start training...
[15:36:12] /home/antholzer/mxnet/dmlc-core/include/dmlc/./logging.h:304: [15:36:12] src/storage/./pooled_storage_manager.h:84: cudaMalloc failed: out of memory
```

Now if I run the same Python variant ([alexnet.py](https://gist.github.com/hesseltuinhof/01d52e5ba64546bf6b806b3ffcc10c3f#file-alexnet-py), [main.py](https://gist.github.com/hesseltuinhof/01d52e5ba64546bf6b806b3ffcc10c3f#file-main-py)), I have no problems with memory. With a batch size of 128 I am running at 3gb memory and with batch size of 256 at around 4gb.

*Note:* At least I was able to train the Julia variant with a batch size of 16.

I wonder why the Julia variant blows up its memory... :question: Anyone has an idea about this or experiences similar issues?