FluxML / FluxML/NNlib.jl

`relu` propagates NaN on CPU but not on GPU

Open
#509 8 comments 1 reaction 0 assignees View on GitHub
bug CUDA good first issue help wanted
Dominant language
Julia
Stars
258
Forks
138
Avg merge
2d 1h
Merged PRs (30d)
3

Description

edit: see https://github.com/FluxML/NNlib.jl/issues/509, this is due to the `relu`, not really the `BatchNorm`

```julia
julia> using Flux

julia> layer = BatchNorm(32, relu)
BatchNorm(32, relu) # 64 parameters, plus 64 non-trainable

julia> layer(NaN32*zeros(Float32, (32,1)))
32×1 Matrix{Float32}:
NaN
NaN
NaN
NaN
NaN
NaN
NaN

NaN
NaN
NaN
NaN
NaN
NaN

julia> gpu(layer)(gpu(NaN32*zeros(Float32, (32,1))))
32×1 CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0

0.0
0.0
0.0
0.0
0.0
0.0
```

edit: just saw I'm on Flux v0.12.10 so maybe this is outdated

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the CPU and GPU examples using BatchNorm(32, relu) with NaN32 input, then inspect the relu implementation and its backend-specific behavior. Done means relu handles NaN consistently on CPU and GPU, with tests covering the reported discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.