clab / clab/dynet

layer_norm for 2D and more?

Open
#1,066 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

Hi all,

Is it useful to have a **layer_norm_2d** (the input is a matrix) in addition to layer_norm?
Currently, I tried a naive version but it may be slow.

```
Expression layer_norm_2d(const Expression& x, const Expression& g, const Expression& b){
std::vector vCols(x.dim().d[1]);
for (unsigned i = 0; i < x.dim().d[1]; i++){
Expression c_x = select_cols(x, {i});
vCols[i] = layer_norm(c_x, g, b);
}
return concatenate_cols(vCols);
}
```
Any suggestion? I would be happy to work on this!

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing layer_norm entry point and the related select_cols and concatenate_cols operations shown in the issue. Determine the intended normalization dimensions and parameter shapes for matrix input, then compare the result and performance with the proposed column-wise approach. Done means the behavior and API are agreed on and covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.