llvm / llvm/torch-mlir

Front-end conversion for nn.Linear() and rearrange() function generate wrong types

Open
#2,115 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.9k
Forks
736
Avg merge
5d 22h
Merged PRs (30d)
15

Description

I write a simple attention model:

```python
from torch import nn
from einops import rearrange
class Attention(nn.Module):
def __init__(self, dim = 32, heads = 3, dim_head = 8):
super().__init__()
inner_dim = dim_head * heads
self.heads = heads
self.scale = dim_head ** -0.5
self.norm = nn.LayerNorm(dim)
self.attend = nn.Softmax(dim = -1) # the last dim

self.to_qkv = nn.Linear(dim, inner_dim * 3, bias = False)
self.to_out = nn.Linear(inner_dim, dim, bias = False)

def forward(self, x):
x = self.norm(x)
q, k, v = self.to_qkv(x).chunk(3, dim = -1)
q, k, v = [einops.rearrange(t, 'n (h d) -> h n d', h=self.heads) for t in (q, k, v)]
dots = torch.matmul(q, k.transpose(-1, -2)) * self.scale
attn = self.attend(dots)
out = torch.matmul(attn, v)
out = rearrange(out, ' h n d -> n (h d)')
return self.to_out(out)

a = Attention(8, heads=1, dim_head = 2)
img2 = torch.ones(16, 8)
preds = a(img2)

module = torch_mlir.compile(a, img2, output_type="tosa",use_tracing=True,verbose=True)
print("TORCH OutputType\n", module.operation.get_asm(large_elements_limit=10))
```

torch_mlir.compile() goes wrong with following infomation:
/
Traceback (most recent call last):
File "/home/tianyi/Torchmlir/Models/vit-pytorch/OnlyAttention.py", line 66, in
module = torch_mlir.compile(a, img2, output_type="tosa",use_tracing=True,verbose=True)
File "/home/tianyi/anaconda3/envs/local-torch-mlir/lib/python3.10/site-packages/torch_mlir/__init__.py", line 391, in compile
run_pipeline_with_repro_report(
File "/home/tianyi/anaconda3/envs/local-torch-mlir/lib/python3.10/site-packages/torch_mlir/compiler_utils.py", line 75, in run_pipeline_with_repro_report
raise TorchMlirCompilerError(trimmed_message) from None
torch_mlir.compiler_utils.TorchMlirCompilerError: Lowering TorchScript IR -> Torch Backend IR failed with the following diagnostics:

python exception: Failure while executing pass pipeline:
error: "/home/tianyi/Torchmlir/Models/vit-pytorch/OnlyAttention.py":44:0: unsupported by backend contract: Unimplemented operator 'aten.chunk'
note: "/home/tianyi/Torchmlir/Models/vit-pytorch/OnlyAttention.py":44:0: see current operation: %21 = "torch.operator"(%20, %13, %14) {name = "aten.chunk"} : (!torch.tensor<[16,6],f32>, !torch.int, !torch.int) -> !torch.list

For Torch-MLIR developers, the error can be reproduced with:
$ torch-mlir-opt -pass-pipeline='builtin.module(torchscript-module-to-torch-backend-pipeline{backend-legal-ops=aten.flatten.using_ints,aten.native_layer_norm,aten.linear extra-library=})' /tmp/Attention.mlir
Add '-mlir-print-ir-after-all -mlir-disable-threading' to get the IR dump for debugging purpose.
/

Here I attach the /tmp/Attention.mlir.
The two wrong operations in Attention.mlir are :

1: torch.aten.linear generated from nn.Linear()
```mlir
func.func private @__torch__.torch.nn.modules.linear.Linear.forward(%arg0: !torch.nn.Module<"__torch__.torch.nn.modules.linear.Linear"> loc(unknown), %arg1: !torch.tensor loc(unknown)) -> !torch.tensor {
%9 = torch.tensor_static_info_cast %arg1 : !torch.tensor to !torch.tensor<[16,8],f32> loc(#loc)
%10 = torch.prim.GetAttr %arg0["weight"] : !torch.nn.Module<"__torch__.torch.nn.modules.linear.Linear"> -> !torch.tensor loc(#loc)
%none_0 = torch.constant.none loc(#loc)
%11 = torch.aten.linear %9, %10, %none_0 : !torch.tensor<[16,8],f32>, !torch.tensor, !torch.none -> !torch.tensor<[16,6],f32> loc(#loc2)
%12 = torch.tensor_static_info_cast %11 : !torch.tensor<[16,6],f32> to !torch.tensor loc(#loc2)
return %12 : !torch.tensor loc(#loc)
} loc(#loc)
```

2: torch.prim.NumToTensor.Scalar generated from einops.rearrange()
```mlir
......
%18:3 = torch.prim.ListUnpack %17 : !torch.list -> !torch.tensor<[16,2],f32>, !torch.tensor<[16,2],f32>, !torch.tensor<[16,2],f32> loc(#loc)
%int0 = torch.constant.int 0 loc(#loc5)
%19 = torch.aten.size.int %18#0, %int0 : !torch.tensor<[16,2],f32>, !torch.int -> !torch.int loc(#loc5)
%20 = torch.prim.NumToTensor.Scalar %19 : !torch.int -> !torch.tensor<[],si64> loc(#loc)
.....
```

How can I fix front end conversion for these two operations? Thanks!
[Attention.mlir.txt](https://github.com/llvm/torch-mlir/files/11454434/Attention.mlir.txt)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied Attention.py reproducer and /tmp/Attention.mlir, then inspect the torch-mlir-opt backend pipeline shown in the error. Trace the aten.linear and torch.prim.NumToTensor.Scalar operations from the generated IR and compare them with the reported aten.chunk failure; done means the attention model converts without the incorrect operations or backend error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.