meta-pytorch / meta-pytorch/KernelAgent
Wrong solution to L3-31_VisionAttention in the artifact repo
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 556
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
Dear KernelAgent team,
Thanks for sharing your work with the community. After carefully examining some results from the artifact reposiroty, I found that the solution to 31_VisionAttention of KernelBench Level 3 might be wrong.
https://github.com/Laurawly/kernelagent-artifacts/blob/main/L3/31_VisionAttention/final_kernel.py
The solution does not implement the attention mechanism. It only implements LayerNorm, but it passes the "wrong test" anyway.
I have tried to generate a solution using the complex Fuser pipeline, but it also gave me a false success. There's no Triton-based attention in the solution, because the test allows the agent to use torch.bmm() and Tensor.matmul().
Would you release all artifacts for us to better understand the situation? Thanks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting L3/31_VisionAttention/final_kernel.py and reproducing the reported success in the KernelBench Level 3 test. Check how use of torch.bmm() and Tensor.matmul() is permitted, and determine why an implementation that only applies LayerNorm passes. Done means the validation behavior or artifact issue is clearly resolved and the result no longer reports a false success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100