RosettaCommons / RosettaCommons/foundry

RF3 error ConvertTritonGPUToLLVM

Open
#214 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
966
Forks
181
Avg merge
4d 4h
Merged PRs (30d)
2

Description

Hi There
First of all thanks for your great work! I am getting an error when running structure prediction with rf3. I am using this command:
rf3 fold inputs=test_input/test_struct.pdb ckpt_path=./checkpoints/rf3_foundry_01_24_latest_remapped.ckpt out_dir=test_pdb

And this is my error:

Unsupported conversion from bf16 to f16 LLVM ERROR: Unsupported rounding mode for conversion. #blocked = #ttg.blocked<{sizePerThread = [4, 4], threadsPerWarp = [4, 8], warpsPerCTA = [4, 1], order = [1, 0]}> #blocked1 = #ttg.blocked<{sizePerThread = [1, 8], threadsPerWarp = [8, 4], warpsPerCTA = [4, 1], order = [1, 0]}> #blocked2 = #ttg.blocked<{sizePerThread = [8, 1], threadsPerWarp = [8, 4], warpsPerCTA = [1, 4], order = [0, 1]}> #blocked3 = #ttg.blocked<{sizePerThread = [8, 1], threadsPerWarp = [4, 8], warpsPerCTA = [1, 4], order = [0, 1]}> #shared = #ttg.swizzled_shared<{vec = 1, perPhase = 1, maxPhase = 1, order = [1, 0]}> #shared1 = #ttg.swizzled_shared<{vec = 1, perPhase = 1, maxPhase = 1, order = [0, 1]}> #smem = #ttg.shared_memory module attributes {"ttg.num-ctas" = 1 : i32, "ttg.num-warps" = 4 : i32, ttg.target = "cuda:70", "ttg.threads-per-warp" = 32 : i32} { tt.func public @fused_sigmoid_gated_dual_gemm_forward_kernel(%arg0: !tt.ptr<bf16> {tt.divisibility = 16 : i32}, %arg1: !tt.ptr<bf16> {tt.divisibility = 16 : i32}, %arg2: !tt.ptr<bf16> {tt.divisibility = 16 : i32}, %arg3: i32 {tt.divisibility = 16 : i32}, %arg4: i32 {tt.divisibility = 16 : i32}, %arg5: i32 {tt.divisibility = 16 : i32}, %arg6: !tt.ptr<bf16> {tt.divisibility = 16 : i32}) attributes {noinline = false} {
...
{-# external_resources: { mlir_reproducer: { pipeline: "builtin.module(triton-nvidia-mma-lowering, tritongpu-combine-tensor-select-and-if, tritongpu-allocate-warp-groups, convert-scf-to-cf, allocate-shared-memory, triton-tensor-memory-allocation, tritongpu-global-scratch-memory-allocation, convert-triton-gpu-to-llvm{compute-capability=70 ptx-version=84}, canonicalize{ max-iterations=10 max-num-rewrites=-1 region-simplify=normal test-convergence=false top-down=true}, cse, convert-nv-gpu-to-llvm, convert-warp-specialize-to-llvm, canonicalize{ max-iterations=10 max-num-rewrites=-1 region-simplify=normal test-convergence=false top-down=true}, cse, symbol-dce, enable-line-info)", disable_threading: false, verify_each: true } } #-} /work/ps40kyji-RFD3/foundry/.venv/lib/python3.12/site-packages/cuequivariance_ops/triton/gated_gemm_triton.py:19:0: error: Failures have been detected while processing an MLIR pass pipeline /work/ps40kyji-RFD3/foundry/.venv/lib/python3.12/site-packages/cuequivariance_ops/triton/gated_gemm_triton.py:19:0: note: Pipeline failed while executing [ConvertTritonGPUToLLVMon 'builtin.module' operation]: reproducer generated atstd::errs, please share the reproducer above with Triton project.Traceback (most recent call last):
What could cause this issue?
Thanks very much in advance!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the documented rf3 fold command and inspect cuequivariance_ops/triton/gated_gemm_triton.py:19 along with the generated MLIR reproducer. Check the Triton, CUDA, and checkpoint/runtime compatibility involved in ConvertTritonGPUToLLVM. Done means identifying the cause of the bf16-to-f16 conversion failure and documenting a verified resolution or compatibility requirement.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.