[Office] drug-inventory-split-L2-006 未校验输出行内容和指定 sheet 名

中文

我在本地检查 wb-bench-office-v1.0 里的 drug-inventory-split-L2-006 时，发现 rule verifier 对输出内容的检查不完整。

这个任务要求把原始药品出入库明细拆成三个 sheet，其中出库和盘亏记录所在的 sheet 必须叫 出库盘亏。为了确认评分逻辑，我做了一个明显错误的 result.xlsx：三个 sheet 都用了错误名称，表头和 RECORD_ID 分组保持正确，其余九列全部写成 CORRUPTED。

任务自带的 eval_core.py 仍然返回：

{
  "passed_count": 15,
  "total_count": 15,
  "pass_rate": 1.0
}

看了一下评分代码，漏检应该来自两处：

• _match_sheets_by_iou() 只根据 RECORD_ID 集合匹配 sheet，没有检查 sheet 名。
• columns_preserved_all_sheets 只检查表头。其他检查会根据原始输入判断每个 ID 应该分到哪里，但不会比较输出行里的其他字段。

比较直接的修法是按 RECORD_ID 对齐源数据和输出数据，再逐字段比较，日期和数字可以先做格式归一化。另外可以单独检查 出库盘亏 这个 sheet 名，并加一个回归用例：保留正确的 ID 分组，但故意修改其他单元格，verifier 应该拒绝这个文件。

这里只反馈 deterministic rule verifier 的结果。Office track 还有独立的 LLM judge，它可能发现部分损坏，所以这个 issue 不代表最终 combined score 一定也是满分。但 rule verifier 对这个文件给出 15/15，应该不是预期行为。

English

While checking drug-inventory-split-L2-006 in the public wb-bench-office-v1.0 archive, I noticed that the rule verifier does not fully validate the output rows.

The task asks the agent to split the source inventory ledger into three sheets, with the outbound and inventory-loss sheet named 出库盘亏. To check the scoring logic, I created an obviously invalid result.xlsx: all three sheet names were wrong, the headers and RECORD_ID grouping were kept, and every other cell was replaced with CORRUPTED.

The published eval_core.py still returned:

{
  "passed_count": 15,
  "total_count": 15,
  "pass_rate": 1.0
}

The gap appears to come from two places:

• _match_sheets_by_iou() matches sheets only by their RECORD_ID sets and does not check their names.
• columns_preserved_all_sheets checks the headers only. The remaining checks use the source workbook to decide which group each ID belongs to, but do not compare the other output fields with the source row.

One way to fix this would be to align source and output rows by RECORD_ID, then compare the remaining fields after normalising dates and numeric values where needed. The verifier could also check the required 出库盘亏 sheet name directly. A regression case that keeps the correct ID grouping while changing the other cells would cover this failure mode.

This report is limited to the deterministic rule verifier. The Office track also has a separate LLM judge, which may catch some of the corruption, so this does not mean the final combined score is necessarily 1.0. The unexpected part is that the rule verifier gives this workbook 15/15.

复现 / Reproduction

以下步骤使用公开数据包，以及 framework commit b516950be5b56eb3be406c2f76ee1c5111dcb57f。

The steps below use the public dataset archive and framework commit b516950be5b56eb3be406c2f76ee1c5111dcb57f.

TASK=wb-bench-office-v1.0/tasks/drug-inventory-split-L2-006
mkdir -p repro-workspace/output
tar xzf "$TASK/environment/workspace.tar.gz" -C repro-workspace

import json
from pathlib import Path
from openpyxl import Workbook

task = Path("wb-bench-office-v1.0/tasks/drug-inventory-split-L2-006")
gold = json.loads((task / "tests/gold/gold_answer.json").read_text())

wb = Workbook()
wb.remove(wb.active)

for name, key in [
    ("wrong-sheet-1", "sheet1_record_ids"),
    ("wrong-sheet-2", "sheet2_record_ids"),
    ("wrong-sheet-3", "sheet3_record_ids"),
]:
    ws = wb.create_sheet(name)
    ws.append(gold["headers"])
    for record_id in gold[key]:
        ws.append([record_id] + ["CORRUPTED"] * (len(gold["headers"]) - 1))

wb.save("repro-workspace/output/result.xlsx")

docker run --rm \
  -v "$PWD/$TASK/tests:/tests:ro" \
  -v "$PWD/repro-workspace:/workspace:ro" \
  python:3.12-slim \
  sh -lc 'pip install -q "openpyxl>=3.1" &&
          python /tests/grading/eval_core.py /workspace/output/result.xlsx'

Start by reproducing the result with tests/grading/eval_core.py for drug-inventory-split-L2-006, then inspect _match_sheets_by_iou() and columns_preserved_all_sheets. Add regression coverage for wrong sheet names and corrupted non-ID cells while preserving the expected grouping. Done means the reproduced workbook no longer receives 15/15.