huggingface / huggingface/candle

No backward pass for `LayerNorm` if tensor is contiguous and remove_mean is true

Open
#2,977 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

This is similar to #2168, but the root cause appears to be different:

* In #2212, a new `candle_nn::ops::layer_norm` function was added, which uses `apply_op3_no_bwd`.

* In #2213, `LayerNorm`'s forward pass was modified to call `candle_nn::ops::layer_norm` when the tensor is contiguous and `remove_mean` is true.

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace LayerNorm's forward path for contiguous tensors with remove_mean enabled, then inspect candle_nn::ops::layer_norm and its apply_op3_no_bwd usage. Reproduce the missing backward pass and compare with the behavior discussed in issues #2168, #2212, and #2213. Done means this LayerNorm path supports backward computation and has a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.