huggingface / huggingface/candle
No backward pass for `RmsNorm` if tensor is contiguous
Open
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
`RmsNorm` switches to faster implementation if tensor is contiguous:
https://github.com/huggingface/candle/blob/82b641fd2752e3b14db6a9c91faef70e3329f3b5/candle-nn/src/layer_norm.rs#L174-L175
But it does not support backward pass:
https://github.com/huggingface/candle/blob/82b641fd2752e3b14db6a9c91faef70e3329f3b5/candle-nn/src/ops.rs#L640
Maybe it's better to implement `ModuleT` rather than `Module` for `RmsNorm` and use faster implementation only if `train == false`?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.