flagos-ai / flagos-ai/FlagTree

[Bug Report] HSACOError & FileNotFoundError when compiling Triton kernel on Hygon BW1000 HCU

Open
#820 2 comments 0 reactions 0 assignees View on GitHub
hcu
Dominant language
Python
Stars
350
Forks
149
Avg merge
2d 4h
Merged PRs (30d)
87

Description

Environment Information
- Image: flagtree-hcu3.6-py310-torch2.4.1-ubuntu22.04:202604-base
- Hardware: Hygon BW1000 (arch: gfx936)
- GPU Count: 8
- Installation Method: Source-free installation following documents/install_hcu_cn.md (Section 2.1)
- Component Versions:
- flagtree: 0.6.0+hcu3.6
- triton: 3.6.0
- TRITON_HIP_CLANG_PATH: /opt/dtk/dcc/bin/clang (clang-17.0.0 from DTK driverruntime branch)
Problem Description

Reproduction Steps
# Launch container
docker run ... flagtree-hcu3.6-py310-torch2.4.1-ubuntu22.04:202604-base

# Inside container (pip install can be skipped as the image is pre-installed)
pip install flagtree===0.6.0+hcu3.6
python3 python/tutorials/01-vector-add.py
Error Log (Key Part)
File ".../triton/backends/hcu/compiler_hcu.py", line 756, in make_amdgcn
with open(asm_file, "r") as fd_out:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpXXXX.amdgcn'
...
HSACOError: HSACO error: File operation failed: ...

Clang stderr output (unknown arguments):
clang (LLVM option parsing): Unknown command line argument '-check-valu-data-forward-hazards=0'
clang (LLVM option parsing): Unknown command line argument '-disable-cluster-lds-memops=true'

Root Cause Analysis
1. The make_amdgcn function of HCU backend passes several HCU-proprietary LLVM flags via _get_clang_args when compiling LLVM IR (.ll) to amdgcn assembly (.amdgcn). These flags are only supported by Hygon-customized LLVM, not standard upstream clang.
2. The path_to_rocm_clang() resolves to /opt/dtk/dcc/bin/clang (standard clang-17), which cannot recognize the HCU-specific flags, leading to compilation failure and missing .amdgcn output file. This triggers FileNotFoundError and is finally wrapped into HSACOError.
3. The default expected clang path for path_to_rocm_clang() is /opt/dtk/aillvm/bin/clang-18 (aillvm branch), which does not exist in the current image.
4. The built-in hcu-llvm22 (~/.flagtree/hcu/hcu-llvm22-xxx) only provides backend tools (llc/opt/lld/mlir-opt) without clang frontend. Itsllc also lacks support for the above HCU-specific flags and is only used for build-time processing, not runtime kernel compilation.

Key Verification Evidence
1. Standard clang-17 has no HCU proprietary flag support:
strings /opt/dtk/dcc/bin/clang-17 | grep -c "check-valu-data-forward-hazards"
# Output: 0
2. Built-in hcu-llvm22 llc lacks all related HCU flags:
llc --help-hidden | grep -E "check-valu-data-forward-hazards|disable-cluster-lds-memops|hcu-pre-emit-load-store-opt|support-768-vgprs"
# No output
3. Expected aillvm clang directory missing:
ls /opt/dtk/aillvm/bin/clang-18
# ls: cannot access '/opt/dtk/aillvm/bin/clang-18': No such file or directory

Questions for Maintainers
- Is the 202604-base image missing the Hygon-customized clang (aillvm clang-18) required for runtime Triton kernel compilation?
- What is the correct compiler toolchain configuration for running Triton HCU kernels on the current official image?
- Is there a temporary workaround to bypass the unsupported hardcoded HCU compilation flags for standard clang?
- Will the official base image be updated to include the complete customized LLVM/clang toolchain for HCU backend?

Contributor guide

Open the contributing guide

Research direction

Start with triton/backends/hcu/compiler_hcu.py, especially make_amdgcn and path_to_rocm_clang, then reproduce python3 python/tutorials/01-vector-add.py in the stated container. Compare the configured clang path and supported flags with the error output; done means Triton HCU kernel compilation completes without the missing .amdgcn or HSACOError.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.