Tencent / Tencent/workbuddy-bench
[WB-Bench-Office] ticket-weekly-L3-010 verifier rejects valid ordinal week labels and summary rows
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 356
- Forks
- 38
- PR merge metrics
- No merged PRs in 30d
Description
Summary
The deterministic verifier for ticket-weekly-L3-010 rejects semantically
correct workbooks when the four requested weekly buckets are labeled
ordinally (第1周 through 第4周), even when each row also contains the correct
Monday and Sunday dates and all numerical results match the gold answer.
It also counts a conventional 合计 row as a fifth week.
In observed outputs, changing only the representation of the week column from
ordinal labels to Monday dates changed the Rule score from 2/15 (0.1333) to
14/15 (0.9333). The underlying four weekly aggregates were identical.
Task instruction
The task asks:
看一下这份客服工单处理记录,把每天的计划处理量和实际解决量按周汇总(每周从周一算起),我想看每周计划多少、实际完成多少、完成率是多少。输出一个新的xlsx文件。
结果保存为
output/result.xlsx。
The instruction requires Monday-based weekly aggregation, but it does not
require ISO week-number labels, prohibit ordinal labels, or prohibit a total
row.
Reproduction evidence
Output A/B: semantically correct but scored 2/15
Two independent workbooks used this structure:
| 周次 | 周起始日(周一) | 周结束日(周日) | 计划处理量 | 实际解决量 | 完成率 |
|---|---|---|---|---|---|
| 第1周 | 2026-04-06 | 2026-04-12 | 9420 | 6852 | 72.74% |
| 第2周 | 2026-04-13 | 2026-04-19 | 9214 | 7214 | 78.29% |
| 第3周 | 2026-04-20 | 2026-04-26 | 9803 | 7310 | 74.57% |
| 第4周 | 2026-04-27 | 2026-05-03 | 9433 | 7269 | 77.06% |
| 合计 | 37870 | 28645 | 75.64% |
Both received:
Rule reward: 0.1333
Passed: 2 / 15
PASS output_is_valid_xlsx
FAIL has_4_weeks: 行数=5, 期望=4
PASS week_labels_present
FAIL all 12 per-week metric checks: name_match 全部失败
All four plan totals, resolved totals, completion rates, and Monday-Sunday date
ranges match tests/gold/gold_answer.json.
Output C: same aggregates, different week representation, scored 14/15
A third workbook contained the same four aggregates and the same 合计 row,
but placed each Monday date directly in the detected week column:
| 周次(起始周一) | 截止周日 | 计划处理量 | 实际解决量 | 完成率 |
|---|---|---|---|---|
| 2026-04-06 | 2026-04-12 | 9420 | 6852 | 72.74% |
| 2026-04-13 | 2026-04-19 | 9214 | 7214 | 78.29% |
| 2026-04-20 | 2026-04-26 | 9803 | 7310 | 74.57% |
| 2026-04-27 | 2026-05-03 | 9433 | 7269 | 77.06% |
| 合计 | 37870 | 28645 | 75.64% |
This received:
Rule reward: 0.9333
Passed: 14 / 15
FAIL has_4_weeks: 行数=5, 期望=4
PASS all other checks, including all 12 weekly metric checks
This isolates the primary failure to output representation rather than weekly
aggregation correctness.
Root cause
Relevant file:
datasets/wb-bench-office-v1.0/tasks/ticket-weekly-L3-010/tests/grading/eval_core.py
-
_match_agent_rows_to_gold()examines only the selected week column. It
does not use adjacent start/end date columns to establish weekly identity. -
The ordinal fallback is entered only when at least one row has already
matched a gold ISO week:if week_col_name and len(matched_by_name) > 0:With a fully ordinal sequence (
第1周through第4周),
matched_by_nameis empty because the gold ISO weeks are 15 through 18.
Therefore the ordinal fallback never runs. -
has_4_weekschecks the raw number of non-empty rows:len(data_rows) == gold_week_countA standard
合计row is consequently treated as an extra weekly bucket. -
_read_output()assumes the first row of the active sheet is the header.
This may also reject otherwise valid presentation-oriented workbooks that
place a title or note above the data table, although the three outputs above
already reproduce the core issue without relying on that condition.
Expected behavior
The verifier should accept equivalent representations when:
- the workbook contains the four correct Monday-Sunday weekly buckets;
- plan totals, resolved totals, and completion rates match the expected values;
- week identity is recoverable from an ISO week label, an ordinal label plus
valid ordered date ranges, or a Monday start date; and - additional presentation rows such as
合计do not alter the weekly data.
For Output A/B above, the four weekly rows should receive credit for all 12
weekly metric checks.
Suggested fix
- Detect and exclude recognized summary rows such as
合计,总计, and
totalbefore enforcingweek_count. - Match weekly rows using a representation-tolerant sequence:
- explicit ISO week labels;
- Monday start dates or date ranges from any suitable date columns;
- ordinal labels when there are exactly
Nordered weekly rows and their
date ranges validate the Monday-based sequence.
- Do not require a partial ISO match before enabling ordinal fallback.
- Optionally locate the actual table header within the first few rows rather
than unconditionally treating row 1 as the header. - Add regression fixtures for both Output A/B and Output C.
Evaluation impact
This is a verifier false negative: two semantically correct outputs received
0.1333 while a representation-equivalent output received 0.9333.
Because changing the verifier changes benchmark scores, the fix should be
versioned and applied consistently to baseline/oracle validation and all
submitted runs. Existing results should not be selectively patched for this
single model or trial.
Environment
- Dataset:
wb-bench-office-v1.0 - Task:
ticket-weekly-L3-010
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with datasets/wb-bench-office-v1.0/tasks/ticket-weekly-L3-010/tests/grading/eval_core.py, especially _match_agent_rows_to_gold() and _read_output(). Reproduce the ordinal-label and Monday-date outputs, then add regression fixtures for both representations. Done means four valid weekly rows receive metric credit, recognized summary rows do not count as weeks, and equivalent valid workbooks score consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100