facebookresearch / facebookresearch/fvcore

FlopCountAnalysis issues

Open
#153 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
236
PR merge metrics
No merged PRs in 30d

Description

i use bellow code to test mha flops. but it's same flops and params when nhead=4 or 8
```
import torch
import torch.nn as nn

class MHAModel(nn.Module):
def __init__(self, dim, nhead, dropout):
super(MHAModel, self).__init__()

self.mha = nn.MultiheadAttention(dim_out, nhead, dropout=dropout, batch_first=True)

def forward(self, x):
x = self.mha(x, x, x)[0]
return x

from fvcore.nn import FlopCountAnalysis, flop_count_table

dim_out = 448
seq_len = 300
nhead = 4
dropout = 0.1

net = MHAModel(dim=dim_out, nhead=nhead, dropout=dropout)
net.eval()
data = torch.randn((1, seq_len, dim_out))
flops = FlopCountAnalysis(net, (data))
print(flop_count_table(flops, max_depth=4))

```

![image](https://github.com/user-attachments/assets/1b80d278-1830-4adb-9f38-923b14acd558)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.