JuliaSIMD / JuliaSIMD/LoopVectorization.jl
`@avxt` harms the performence of `.Threads`
- Dominant language
- Julia
- Stars
- 789
- Forks
- 73
- PR merge metrics
- No merged PRs in 30d
Description
I have tried to replace `@avx` my framework with `@avxt` since it should have a better performence. However, the neural network has become unbelievably slow. So I did a small experiment with FCNN which almost only requires matmul and matadd.
The structure of network:
```julia
model = Sequential()
model.add_layer(model, Dense; input_size=786, layer_size=512, activation_function=ReLU)
model.add_layer(model, Dense; layer_size=256, activation_function=ReLU)
model.add_layer(model, Dense; layer_size=128, activation_function=ReLU)
model.add_layer(model, Dense; layer_size=64, activation_function=ReLU)
model.add_layer(model, Dense; layer_size=10, activation_function=Softmax_CEL)
SGD.fit(model=model, input_data=flatten(train_x, 3), output_data=One_Hot(train_y, 10, dict),
loss_function=Categorical_Cross_Entropy_Loss, monitor=Classification, epochs=10, batch=128)
```
The code using `@avx` or `@avxt`:
```julia
function activate_Dense(layer::Dense, input::Array{Float32})
@avxt for x in axes(layer.weights, 1), y in axes(input, 2)
c = 0.0f0
for z in axes(layer.weights, 2)
c += layer.weights[x,z]*input[z,y]
end
layer.value[x,y] = c+layer.biases[x]
end
# layer.value = layer.weights*input .+ layer.biases
@time layer.output = layer.activation_function.func(layer.value)
end
function update_Dense(layer::Dense, optimizer::String, Last_Layer_output::Array{Float32}, Next_Layer_propagation_units::Array{Float32}, α::Float64, parameters::Tuple, direction::Int64=1)
@time layer.activation_function.get_∇biases!(layer.∇biases, layer.value, Next_Layer_propagation_units)
@avxt for x in axes(layer.weights, 2), y in axes(layer.∇biases, 2)
c = 0.0f0
for z in axes(layer.weights, 1)
c += layer.weights[z,x]*layer.∇biases[z,y]
end
layer.propagation_units[x,y] = c
end
# layer.propagation_units = transpose(layer.weights)*∇biases
@time if optimizer=="SGD"
@avxt for x in axes(layer.∇biases, 1), y in axes(Last_Layer_output, 1)
layer.weights[x,y] -= α*layer.∇biases[x,1]*Last_Layer_output[y,1]*direction
end
# layer.weights -= ∇biases*transpose(Last_Layer_output).*α
@avxt for i in 1:length(layer.biases)
layer.biases[i] -= α*layer.∇biases[i,1]*direction
end
# layer.biases -= sum(∇biases, dims=2).*α
println()
end
end
```
I measured the time of parts using mutithreading and the result is quite interesting. Time with `@avxt`:
```julia
# time for forward propagation
0.039304 seconds (44 allocations: 8.969 KiB)
0.039106 seconds (44 allocations: 7.984 KiB)
0.000073 seconds (44 allocations: 7.453 KiB)
0.000015 seconds (43 allocations: 7.141 KiB)
0.000021 seconds (88 allocations: 14.203 KiB)
# backpropagation of layer 5
0.000011 seconds (48 allocations: 7.656 KiB)
# backpropagation of layer 4
0.000028 seconds (41 allocations: 6.781 KiB)
# backpropagation of layer 3
0.038863 seconds (42 allocations: 6.812 KiB)
# backpropagation of layer 2
0.038591 seconds (43 allocations: 6.844 KiB)
# backpropagation of layer 1
0.038039 seconds (43 allocations: 6.844 KiB)
```
Time with `@avx`:
```julia
# time for forward propagation
0.000013 seconds (43 allocations: 8.938 KiB)
0.000012 seconds (43 allocations: 7.953 KiB)
0.000018 seconds (43 allocations: 7.422 KiB)
0.000011 seconds (43 allocations: 7.141 KiB)
0.000020 seconds (88 allocations: 14.203 KiB)
# backpropagation of layer 5
0.000012 seconds (48 allocations: 7.656 KiB)
# backpropagation of layer 4
0.000009 seconds (41 allocations: 6.781 KiB)
# backpropagation of layer 3
0.000009 seconds (41 allocations: 6.781 KiB)
# backpropagation of layer 2
0.000010 seconds (41 allocations: 6.781 KiB)
# backpropagation of layer 1
0.000010 seconds (41 allocations: 6.781 KiB)
```
It seems that `@avxt` is competing against multithreading of base even it is not called.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.