Using ReverseDiff as a Knet backend instead of AutoGrad

Hi,

I ported Knet's MNIST example. You can see changes I made by comparing current reversediff branch with the revision two commits before.

Speed Issues

Although I am taking advantage of ReverseDiff's tape compilation feature, currently ReverseDiff is too slow compared to AutoGrad on this MLP example. Here are the results,

60.575495 seconds (1.21 M allocations: 2.609 GB, 0.44% gc time) (ReverseDiff.jl - compiled)
32.504943 seconds (5.49 M allocations: 6.813 GB, 1.42% gc time) (AutoGrad.jl)

Capabilities

• ReverseDiff has not the ability to work with KnetArrays.
• In Knet, we use ReLU activation (max(0,x) actually), but ReverseDiff is not able to take derivative of this operation currently.
• In AutoGrad, we have a loss function and its first input parameter is weights bundle. It can be an array, a tuple, a dictionary or a combination of these structures. Actually, this is good, because we can use same loss function for different networks (e.g. 1 hidden layer MLP and 2 hidden layer MLP, both uses same loss function). Unlike AutoGrad, in ReverseDiff, we need to pass all parameters to the loss function.
• I think indexing does not work for ReverseDiff. In neural networks, we heavily take advantage of indexing which brings both speed and memory improvements (the old method we were using was matrix multiplication with one-hot vectors). This is what I'm talking about,

julia> using AutoGrad

julia> using ReverseDiff

julia> using ReverseDiff: gradient

julia> f(x,y,i) = sumabs2(x[i]-y)
f (generic function with 1 method)

julia> gradient(f, (rand(3,4),1,1))
ERROR: MethodError: objects of type Int64 are not callable
 in Type at /mnt/kufs/scratch/ikesen16/.julia/somon/v0.5/ReverseDiff/src/api/Config.jl:46 [inlined]
 in Type at /mnt/kufs/scratch/ikesen16/.julia/somon/v0.5/ReverseDiff/src/api/Config.jl:37 [inlined] (repeats 2 times)
 in gradient(::Function, ::Tuple{Array{Float64,2},Int64,Int64}) at /mnt/kufs/scratch/ikesen16/.julia/somon/v0.5/ReverseDiff/src/api/gradients.jl:22

julia> gf = grad(f)
(::gradfun) (generic function with 1 method)

julia> gf(rand(3,4),1,1)
3×4 Array{Float64,2}:
 -0.916041  0.0  0.0  0.0
  0.0       0.0  0.0  0.0
  0.0       0.0  0.0  0.0

• Optional arguments is not supported by ReverseDiff.
• In softmax operation, we have a safer version which prevents float overflow and it takes advantage of maximum operation. However, ReverseDiff does not support maximum/minimum functions.

Begin with the linked Knet MNIST example and reproduce the ReverseDiff-versus-AutoGrad timing and indexing failure. Read src/api/Config.jl and src/api/gradients.jl, then verify each listed limitation—KnetArrays, ReLU, argument structures, optional arguments, and max/min—against focused tests; done requires an agreed scope and passing tests for the selected capability.