JuliaML / JuliaML/MLUtils.jl

Re-export normalise from NNlib to remove duplication

Open
#254 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
124
Forks
23
PR merge metrics
No merged PRs in 30d

Description

### Proposal

MLUtils and NNlib both define a `normalise(x; dims, ...)` (z-score standardization). Since MLUtils already depends on NNlib, MLUtils could drop its own definition and instead re-export `NNlib.normalise`, removing the duplication.

### Current state

- `MLUtils.normalise(x; dims=ndims(x), ϵ=ofeltype(x, 1e-5))` — [src/utils.jl](https://github.com/JuliaML/MLUtils.jl/blob/main/src/utils.jl)
- `NNlib.normalise(x; dims=ndims(x), eps=1f-5)` — added to NNlib in the normalization work ([FluxML/NNlib.jl#748](https://github.com/FluxML/NNlib.jl/pull/748)); it is the stateless building block behind `NNlib.layernorm`.

### The one difference to decide on

The two implementations place `eps`/`ϵ` differently:

| | formula | eps added to |
|---|---|---|
| `MLUtils.normalise` | `(x - μ) / (std + ϵ)` | standard deviation (outside sqrt) |
| `NNlib.normalise` | `(x - μ) / sqrt(var + eps)` | variance (inside sqrt) |

NNlib's inside-the-sqrt convention matches `layernorm`/cuDNN. Re-exporting would therefore be a (small) behavior change for MLUtils users, plus a keyword rename `ϵ` → `eps`. Both seem acceptable given the tiny numerical impact, but flagging it explicitly.

Also worth noting: MLUtils' current implementation carries a stale `# use this when Zygote#478 gets merged` comment and computes `std` without passing `mean=μ`; NNlib passes `mean=μ` to `var`, which is correct at first *and* second order (verified: Hessian is bit-identical to the recompute form).

### Options

1. Re-export `NNlib.normalise` (accept eps-placement change + `ϵ`→`eps` rename).
2. Keep a thin `MLUtils.normalise` that forwards to NNlib but preserves the `ϵ` outside-sqrt semantics for backward compat.
3. Leave as-is.

Happy to open a PR for whichever direction is preferred.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with MLUtils' implementation in src/utils.jl and compare it with NNlib.normalise from the normalization work in FluxML/NNlib.jl#748. Resolve whether to re-export, preserve MLUtils' existing semantics through a forwarding wrapper, or leave the duplication; done means the chosen eps behavior and keyword compatibility are explicit and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
machine-learning
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.