JuliaSIMD / JuliaSIMD/LoopVectorization.jl

`@avxt` harms the performence of `.Threads`

Open
#221 18 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
789
Forks
73
PR merge metrics
No merged PRs in 30d

Description

I have tried to replace `@avx` my framework with `@avxt` since it should have a better performence. However, the neural network has become unbelievably slow. So I did a small experiment with FCNN which almost only requires matmul and matadd.

The structure of network:
```julia
model = Sequential()
model.add_layer(model, Dense; input_size=786, layer_size=512, activation_function=ReLU)
model.add_layer(model, Dense; layer_size=256, activation_function=ReLU)
model.add_layer(model, Dense; layer_size=128, activation_function=ReLU)
model.add_layer(model, Dense; layer_size=64, activation_function=ReLU)
model.add_layer(model, Dense; layer_size=10, activation_function=Softmax_CEL)

SGD.fit(model=model, input_data=flatten(train_x, 3), output_data=One_Hot(train_y, 10, dict),
loss_function=Categorical_Cross_Entropy_Loss, monitor=Classification, epochs=10, batch=128)
```

The code using `@avx` or `@avxt`:
```julia
function activate_Dense(layer::Dense, input::Array{Float32})
@avxt for x in axes(layer.weights, 1), y in axes(input, 2)
c = 0.0f0
for z in axes(layer.weights, 2)
c += layer.weights[x,z]*input[z,y]
end
layer.value[x,y] = c+layer.biases[x]
end
# layer.value = layer.weights*input .+ layer.biases
@time layer.output = layer.activation_function.func(layer.value)
end

function update_Dense(layer::Dense, optimizer::String, Last_Layer_output::Array{Float32}, Next_Layer_propagation_units::Array{Float32}, α::Float64, parameters::Tuple, direction::Int64=1)
@time layer.activation_function.get_∇biases!(layer.∇biases, layer.value, Next_Layer_propagation_units)
@avxt for x in axes(layer.weights, 2), y in axes(layer.∇biases, 2)
c = 0.0f0
for z in axes(layer.weights, 1)
c += layer.weights[z,x]*layer.∇biases[z,y]
end
layer.propagation_units[x,y] = c
end
# layer.propagation_units = transpose(layer.weights)*∇biases

@time if optimizer=="SGD"
@avxt for x in axes(layer.∇biases, 1), y in axes(Last_Layer_output, 1)
layer.weights[x,y] -= α*layer.∇biases[x,1]*Last_Layer_output[y,1]*direction
end
# layer.weights -= ∇biases*transpose(Last_Layer_output).*α
@avxt for i in 1:length(layer.biases)
layer.biases[i] -= α*layer.∇biases[i,1]*direction
end
# layer.biases -= sum(∇biases, dims=2).*α
println()
end
end
```

I measured the time of parts using mutithreading and the result is quite interesting. Time with `@avxt`:
```julia
# time for forward propagation
0.039304 seconds (44 allocations: 8.969 KiB)
0.039106 seconds (44 allocations: 7.984 KiB)
0.000073 seconds (44 allocations: 7.453 KiB)
0.000015 seconds (43 allocations: 7.141 KiB)
0.000021 seconds (88 allocations: 14.203 KiB)

# backpropagation of layer 5
0.000011 seconds (48 allocations: 7.656 KiB)

# backpropagation of layer 4
0.000028 seconds (41 allocations: 6.781 KiB)

# backpropagation of layer 3
0.038863 seconds (42 allocations: 6.812 KiB)

# backpropagation of layer 2
0.038591 seconds (43 allocations: 6.844 KiB)

# backpropagation of layer 1
0.038039 seconds (43 allocations: 6.844 KiB)
```

Time with `@avx`:
```julia
# time for forward propagation
0.000013 seconds (43 allocations: 8.938 KiB)
0.000012 seconds (43 allocations: 7.953 KiB)
0.000018 seconds (43 allocations: 7.422 KiB)
0.000011 seconds (43 allocations: 7.141 KiB)
0.000020 seconds (88 allocations: 14.203 KiB)

# backpropagation of layer 5
0.000012 seconds (48 allocations: 7.656 KiB)

# backpropagation of layer 4
0.000009 seconds (41 allocations: 6.781 KiB)

# backpropagation of layer 3
0.000009 seconds (41 allocations: 6.781 KiB)

# backpropagation of layer 2
0.000010 seconds (41 allocations: 6.781 KiB)

# backpropagation of layer 1
0.000010 seconds (41 allocations: 6.781 KiB)
```

It seems that `@avxt` is competing against multithreading of base even it is not called.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.