instadeepai / instadeepai/fastpbrl
Question about LayerNorm in VectorizedLinearLayer
- Dominant language
- Python
- Stars
- 58
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
You implement LayerNorm in VectorizedLinearLayer but the LayerNorm input is "self._out_features, self._population_size", which means that the eps in LayerNorm is self._population_size. Could you please explain why ?
```python
self._layer_norm = (
torch.nn.LayerNorm(self._out_features, self._population_size)
if use_layer_norm
else None
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
assert x.shape[0] == self._population_size
if self._layer_norm is not None:
return self._layer_norm(x.matmul(self.weight) + self.bias)
return x.matmul(self.weight) + self.bias
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating VectorizedLinearLayer and inspect the torch.nn.LayerNorm constructor call and forward path shown in the issue. Determine whether the reported argument is intentional, then document the rationale or identify the correction; no test or file path is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100