meta-pytorch / meta-pytorch/KernelAgent

Wrong solution to L3-31_VisionAttention in the artifact repo

Open
#52 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
556
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Dear KernelAgent team,

Thanks for sharing your work with the community. After carefully examining some results from the artifact reposiroty, I found that the solution to 31_VisionAttention of KernelBench Level 3 might be wrong.

https://github.com/Laurawly/kernelagent-artifacts/blob/main/L3/31_VisionAttention/final_kernel.py

The solution does not implement the attention mechanism. It only implements LayerNorm, but it passes the "wrong test" anyway.

I have tried to generate a solution using the complex Fuser pipeline, but it also gave me a false success. There's no Triton-based attention in the solution, because the test allows the agent to use torch.bmm() and Tensor.matmul().

Would you release all artifacts for us to better understand the situation? Thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting L3/31_VisionAttention/final_kernel.py and reproducing the reported success in the KernelBench Level 3 test. Check how use of torch.bmm() and Tensor.matmul() is permitted, and determine why an implementation that only applies LayerNorm passes. Done means the validation behavior or artifact issue is clearly resolved and the result no longer reports a false success.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.