[Bugs][Iluvatar] vllm 在安装了tree的环境中启动时 triton报错了

# [Iluvatar/corex] Triton iluvatar backend 找不到 libdevice —— `FileNotFoundError: /usr/nvvm/libdevice/libdevice.compute_bi.10.bc`

> 提交给天数(Iluvatar)镜像 / Triton iluvatar backend 相关团队。
> 关键结论先行:**镜像里 Triton 的 iluvatar backend 把 libdevice 硬编码到 NVIDIA 风格路径 `/usr/nvvm/libdevice/`,但 corex 4.5.0 实际把 libdevice 装在 `/usr/local/corex-4.5.0/nvvm/libdevice/`**,导致任何 Triton kernel 首次编译时 `file_hash` 打开文件失败,worker 全部初始化失败,`vllm serve` 起不来。

## 问题概述

在 Iluvatar(天数智芯 / corex 4.5.0)镜像上启动 `vllm serve`(TP8,8 卡),引擎 core 启动到 worker 初始化阶段,第一次触发 Triton kernel 编译(FlagGems 的 `empty`/`zeros` 算子,来自 pynccl 通信器里的 `torch.zeros(1)`)时,8 个 worker 进程**同时**抛出 `FileNotFoundError`,报找不到 `/usr/nvvm/libdevice/libdevice.compute_bi.10.bc`,随后 engine core 初始化失败,APIServer 报 `Engine core initialization failed`。

## 环境信息

- **镜像**(docker image tag):
```
harbor.baai.ac.cn/plugin/iluvatar-corex4.5.0-flagtreenone-triton3.2.0-cxnone-plugin0.2.0-vllm0.24.0-cp312-pt210-x64:20250813
```
tag 解读:corex 4.5.0 / flagtree none / triton 3.2.0 / FlagCX none / plugin-FL 0.2.0 / vllm 0.24.0 / py3.12 / torch 2.10
- **硬件**:Iluvatar BI-V150 × 16(本任务用后 8 卡 `CUDA_VISIBLE_DEVICES=8..15`)
- **软件版本**:
- corex:`/usr/local/corex-4.5.0`(另有软链 `/usr/local/corex -> /usr/local/corex-4.5.0/`)
- Triton:3.2.0(注意存在 `site-packages/triton/backends/iluvatar/` 这个 vendor backend)
- FlagGems:`5.3.4.post1.dev11+gbc6d9426c`
- vLLM:`0.24.0+empty`(`VLLM_TARGET_DEVICE=empty` 安装)
- vllm-plugin-FL:`0.2.0+g2fab72442`(editable)
- Python 3.12 / torch 2.10
- **模型**:XingChen4-29B-A4B(BF16,MoE + MLA + mHC + MTP;TP=8)
- **关键路径事实**:
- 报错要找的文件实际存在:`/usr/local/corex-4.5.0/nvvm/libdevice/libdevice.compute_bi.10.bc` ✅
- 报错要的路径 `/usr/nvvm/libdevice/` **不存在**(`ls /usr/nvvm` → No such file or directory)❌

## 复现命令

容器内(镜像 `/workspace` 已带 vllm / vllm-plugin-FL;FlagGems 已 pip 预装):

```bash
# 环境变量(起服务必须)
export GEMS_VENDOR=iluvatar
export VLLM_PLUGINS=fl
export CUDA_VISIBLE_DEVICES=8,9,10,11,12,13,14,15
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_FL_FLAGOS_BLACKLIST=sort,sort_stable
export VLLM_ENGINE_ITERATION_TIMEOUT_S=72000
export VLLM_RPC_TIMEOUT=72000000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200

# 起服务(TP=8)
vllm serve /models/XingChen4-29B-A4B \
--served-model-name xingchen429 \
--dtype bfloat16 \
--tensor-parallel-size 8 \
--max-model-len 32768 \
--gpu-memory-utilization 0.9 \
--port 8077 \
--attention-backend TRITON_MLA \
--chat-template /models/XingChen4-29B-A4B/chat_template.jinja \
--enforce-eager \
--trust-remote-code
```

**最小触发点**:只要任何一次 Triton kernel 编译发生即可复现(不必起完整模型)。上面的 worker traceback 显示第一次编译是 FlagGems `empty`/`zeros` 算子 —— 这是 pynccl 通信器里 `torch.zeros(1, device=device)` 触发的,发生在 `init_worker_distributed_environment` → `GroupCoordinator.__init__` → `PyNcclCommunicator.__init__`。因此一个更小的复现可以是:在容器内 `import flag_gems; import torch; torch.zeros(1, device="cuda")`(需先 `GEMS_VENDOR=iluvatar` 使 FlagGems 走 iluvatar backend),第一次编译 `empty_kernel` 即会触发同样的 `file_hash` 报错。

## 错误日志关键内容

每个 worker(pid=2183..2190,共 8 个)在同一时刻抛出相同的 traceback,核心片段如下:

```
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] File "/usr/local/lib/python3.12/site-packages/flag_gems/ops/empty.py", line 92, in empty
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] empty_kernel[grid_fn](out, N, BLOCK_SIZE=1024)
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] File "/usr/local/lib/python3.12/site-packages/triton/runtime/jit.py", line 682, in run
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] kernel = self.compile(...)
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] File "/usr/local/lib/python3.12/site-packages/triton/compiler/compiler.py", line 231, in compile
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] key = f"{triton_key()}-{src.hash()}-{backend.hash()}-{options.hash()}-{str(sorted(env_vars.items()))}"
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] File "/usr/local/lib/python3.12/site-packages/triton/backends/iluvatar/compiler.py", line 66, in hash
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] hash_dict["extern_libs"] = tuple((k, file_hash(v)) for k, v in sorted(hash_dict["extern_libs"]))
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] File "/usr/local/lib/python3.12/site-packages/triton/backends/iluvatar/compiler.py", line 24, in file_hash
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] with open(path, "rb") as f:
(Worker pid=2183) ERROR 09-03 15:12:49 [multiproc_executor.py:898] FileNotFoundError: [Errno 2] No such file or directory: '/usr/nvvm/libdevice/libdevice.compute_bi.10.bc'
```

随后:

```
(EngineCore pid=2040) ERROR 09-03 15:12:54 [core.py:1231] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(APIServer pid=1750) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
```

## 根因分析

1. Triton 的 iluvatar vendor backend(`site-packages/triton/backends/iluvatar/compiler.py`)在计算编译缓存 key 时,会对 `extern_libs` 里的每个文件调 `file_hash(path)`(`open(path, "rb")`)。
2. `extern_libs` 中 libdevice 的路径被解析成 NVIDIA 风格默认路径 `/usr/nvvm/libdevice/libdevice.compute_bi.10.bc`。
3. 但 corex 4.5.0 镜像里**不存在** `/usr/nvvm`,libdevice 实际位于 `/usr/local/corex-4.5.0/nvvm/libdevice/libdevice.compute_bi.10.bc`(通过软链 `/usr/local/corex` 也可达)。
4. 于是 `open()` 直接 `FileNotFoundError`,且发生在**首次** Triton kernel 编译(编译缓存 key 计算阶段),所以任何用到 Triton kernel 的推理都无法启动 —— 不是偶发,是必现。

**一句话**:iluvatar Triton backend 的 libdevice 默认路径与 corex 4.5.0 实际安装布局不一致(硬编码了 NVIDIA 的 `/usr/nvvm/libdevice`,未跟随 corex 的 `/usr/local/corex*/nvvm/libdevice`)。

## 建议的修复方向(供相关团队参考)

任选其一(或组合):

1. **镜像侧**:补一个软链 `/usr/nvvm -> /usr/local/corex/nvvm`(最省事,不碰 Triton 代码)。
2. **Triton iluvatar backend 侧**:把 libdevice 路径从硬编码 `/usr/nvvm/libdevice/` 改为从 corex 安装前缀解析(如 `$ILUVATAR_HOME` / `/usr/local/corex*/nvvm/libdevice/`),并做存在性校验 + 报错时打印实际搜索路径。
3. **环境变量**:若该 backend 已支持 libdevice 路径覆盖(如 `TRITON_LIBDEVICE_PATH`),在镜像里默认导出到 corex 路径。

## 附件 / 复现产物

- 完整 serve 日志:`/models/day0_logs/xingchen4-iluvatar-serve.log`(容器 `xingchen4` 内,8 个 worker traceback 全量在 316–779 行区间)
- 起服务脚本:`/models/running_common/serve_xingchen4.sh`(容器内)

---

**期望结果**:corex 镜像上首次 Triton kernel 编译能正确定位 libdevice(`/usr/local/corex*/nvvm/libdevice/libdevice.compute_bi.10.bc`),`vllm serve` 能正常完成 worker 初始化并起服务。

Start with triton/backends/iluvatar/compiler.py, especially file_hash and the extern_libs path used during compiler hashing. Run the minimal flag_gems and torch.zeros reproduction with GEMS_VENDOR=iluvatar, then compare the resolved path with the corex libdevice location. Done means the first Triton kernel compilation finds libdevice and the supplied vllm serve command completes worker initialization.

Written by the indexing model from the issue text.