`relu` propagates NaN on CPU but not on GPU
- Dominant language
- Julia
- Stars
- 258
- Forks
- 138
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 3
Description
edit: see https://github.com/FluxML/NNlib.jl/issues/509, this is due to the `relu`, not really the `BatchNorm`
```julia
julia> using Flux
julia> layer = BatchNorm(32, relu)
BatchNorm(32, relu) # 64 parameters, plus 64 non-trainable
julia> layer(NaN32*zeros(Float32, (32,1)))
32×1 Matrix{Float32}:
NaN
NaN
NaN
NaN
NaN
NaN
NaN
⋮
NaN
NaN
NaN
NaN
NaN
NaN
julia> gpu(layer)(gpu(NaN32*zeros(Float32, (32,1))))
32×1 CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
⋮
0.0
0.0
0.0
0.0
0.0
0.0
```
edit: just saw I'm on Flux v0.12.10 so maybe this is outdated
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the CPU and GPU examples using BatchNorm(32, relu) with NaN32 input, then inspect the relu implementation and its backend-specific behavior. Done means relu handles NaN consistently on CPU and GPU, with tests covering the reported discrepancy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100