Tencent / Tencent/workbuddy-bench

[WB-Bench-Office] ticket-weekly-L3-010 verifier rejects valid ordinal week labels and summary rows

Open
#5 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
356
Forks
38
PR merge metrics
No merged PRs in 30d

Description

Summary

The deterministic verifier for ticket-weekly-L3-010 rejects semantically
correct workbooks when the four requested weekly buckets are labeled
ordinally (第1周 through 第4周), even when each row also contains the correct
Monday and Sunday dates and all numerical results match the gold answer.

It also counts a conventional 合计 row as a fifth week.

In observed outputs, changing only the representation of the week column from
ordinal labels to Monday dates changed the Rule score from 2/15 (0.1333) to
14/15 (0.9333). The underlying four weekly aggregates were identical.

Task instruction

The task asks:

看一下这份客服工单处理记录,把每天的计划处理量和实际解决量按周汇总(每周从周一算起),我想看每周计划多少、实际完成多少、完成率是多少。输出一个新的xlsx文件。

结果保存为 output/result.xlsx

The instruction requires Monday-based weekly aggregation, but it does not
require ISO week-number labels, prohibit ordinal labels, or prohibit a total
row.

Reproduction evidence

Output A/B: semantically correct but scored 2/15

Two independent workbooks used this structure:

周次 周起始日(周一) 周结束日(周日) 计划处理量 实际解决量 完成率
第1周 2026-04-06 2026-04-12 9420 6852 72.74%
第2周 2026-04-13 2026-04-19 9214 7214 78.29%
第3周 2026-04-20 2026-04-26 9803 7310 74.57%
第4周 2026-04-27 2026-05-03 9433 7269 77.06%
合计 37870 28645 75.64%

Both received:

Rule reward: 0.1333
Passed: 2 / 15

PASS output_is_valid_xlsx
FAIL has_4_weeks: 行数=5, 期望=4
PASS week_labels_present
FAIL all 12 per-week metric checks: name_match 全部失败

All four plan totals, resolved totals, completion rates, and Monday-Sunday date
ranges match tests/gold/gold_answer.json.

Output C: same aggregates, different week representation, scored 14/15

A third workbook contained the same four aggregates and the same 合计 row,
but placed each Monday date directly in the detected week column:

周次(起始周一) 截止周日 计划处理量 实际解决量 完成率
2026-04-06 2026-04-12 9420 6852 72.74%
2026-04-13 2026-04-19 9214 7214 78.29%
2026-04-20 2026-04-26 9803 7310 74.57%
2026-04-27 2026-05-03 9433 7269 77.06%
合计 37870 28645 75.64%

This received:

Rule reward: 0.9333
Passed: 14 / 15

FAIL has_4_weeks: 行数=5, 期望=4
PASS all other checks, including all 12 weekly metric checks

This isolates the primary failure to output representation rather than weekly
aggregation correctness.

Root cause

Relevant file:

datasets/wb-bench-office-v1.0/tasks/ticket-weekly-L3-010/tests/grading/eval_core.py
  1. _match_agent_rows_to_gold() examines only the selected week column. It
    does not use adjacent start/end date columns to establish weekly identity.

  2. The ordinal fallback is entered only when at least one row has already
    matched a gold ISO week:

    if week_col_name and len(matched_by_name) > 0:
    

    With a fully ordinal sequence (第1周 through 第4周),
    matched_by_name is empty because the gold ISO weeks are 15 through 18.
    Therefore the ordinal fallback never runs.

  3. has_4_weeks checks the raw number of non-empty rows:

    len(data_rows) == gold_week_count
    

    A standard 合计 row is consequently treated as an extra weekly bucket.

  4. _read_output() assumes the first row of the active sheet is the header.
    This may also reject otherwise valid presentation-oriented workbooks that
    place a title or note above the data table, although the three outputs above
    already reproduce the core issue without relying on that condition.

Expected behavior

The verifier should accept equivalent representations when:

  • the workbook contains the four correct Monday-Sunday weekly buckets;
  • plan totals, resolved totals, and completion rates match the expected values;
  • week identity is recoverable from an ISO week label, an ordinal label plus
    valid ordered date ranges, or a Monday start date; and
  • additional presentation rows such as 合计 do not alter the weekly data.

For Output A/B above, the four weekly rows should receive credit for all 12
weekly metric checks.

Suggested fix

  1. Detect and exclude recognized summary rows such as 合计, 总计, and
    total before enforcing week_count.
  2. Match weekly rows using a representation-tolerant sequence:
    • explicit ISO week labels;
    • Monday start dates or date ranges from any suitable date columns;
    • ordinal labels when there are exactly N ordered weekly rows and their
      date ranges validate the Monday-based sequence.
  3. Do not require a partial ISO match before enabling ordinal fallback.
  4. Optionally locate the actual table header within the first few rows rather
    than unconditionally treating row 1 as the header.
  5. Add regression fixtures for both Output A/B and Output C.

Evaluation impact

This is a verifier false negative: two semantically correct outputs received
0.1333 while a representation-equivalent output received 0.9333.

Because changing the verifier changes benchmark scores, the fix should be
versioned and applied consistently to baseline/oracle validation and all
submitted runs. Existing results should not be selectively patched for this
single model or trial.

Environment

  • Dataset: wb-bench-office-v1.0
  • Task: ticket-weekly-L3-010

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with datasets/wb-bench-office-v1.0/tasks/ticket-weekly-L3-010/tests/grading/eval_core.py, especially _match_agent_rows_to_gold() and _read_output(). Reproduce the ordinal-label and Monday-date outputs, then add regression fixtures for both representations. Done means four valid weekly rows receive metric credit, recognized summary rows do not count as weeks, and equivalent valid workbooks score consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.