layer_norm for 2D and more?
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
Hi all,
Is it useful to have a **layer_norm_2d** (the input is a matrix) in addition to layer_norm?
Currently, I tried a naive version but it may be slow.
```
Expression layer_norm_2d(const Expression& x, const Expression& g, const Expression& b){
std::vector vCols(x.dim().d[1]);
for (unsigned i = 0; i < x.dim().d[1]; i++){
Expression c_x = select_cols(x, {i});
vCols[i] = layer_norm(c_x, g, b);
}
return concatenate_cols(vCols);
}
```
Any suggestion? I would be happy to work on this!
Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the existing layer_norm entry point and the related select_cols and concatenate_cols operations shown in the issue. Determine the intended normalization dimensions and parameter shapes for matrix input, then compare the result and performance with the proposed column-wise approach. Done means the behavior and API are agreed on and covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100