microsoft / microsoft/STATE-Bench

State scorer collapses sibling `order_items` (and orders) — `_record_identity` keys child records by their parent id, causing false `task_completion = 0`

Open Beginner friendly
#36 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
91
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Summary

evaluate_state_requirements reconstructs the final DB snapshot from the task-env
file, which stores entities as lists. scoring.py::_record_identity derives a
record's key from the first matching id field, but it checks owner/foreign keys
(order_id, customer_id, cart_id) before the record's own id (item_id,
…). An order_item carries both item_id and its parent order_id, so all line
items in the same order collapse under one key — only the last survives and its
siblings disappear from the reconstructed snapshot. Their state_requirements
then report as missing_assertions, failing the task even when the agent behaved
perfectly.

Environment

  • STATE-Bench v0.7.1 (commit a0ffc65)
  • state_bench/scoring.py, function _record_identity (around line 206)

Reproduction

Run from a STATE-Bench checkout (uses only shipped files — task
customer_support/13-cancel_partial):

import json
from pathlib import Path
from state_bench.scoring import _record_identity, evaluate_state_requirements
from state_bench.schemas import TaskDefinition, StateDiff

# Root cause: three sibling order_items collapse to one key.
items = [{"item_id": f"ITEM-{n}", "order_id": "ORD-1", "item_status": "confirmed"} for n in (1, 2, 3)]
print([_record_identity(i) for i in items])   # -> ['ORD-1', 'ORD-1', 'ORD-1']

# End-to-end: a correct partial cancellation of only the shirt (ITEM-9105).
task = TaskDefinition.from_dict(json.loads(
    Path("state_bench/domains/customer_support/tasks/13-cancel_partial.json").read_text()))
state_diff = StateDiff(modified={
    "orders": {"ORD-6016": {"status": {"old": "processing", "new": "partially_cancelled"}}},
    "order_items": {"ITEM-9105": {
        "item_status": {"old": "confirmed", "new": "cancelled"},
        "refund_amount": {"old": None, "new": 59},
        "refund_method": {"old": None, "new": "credit_card"},
    }},
})
res = evaluate_state_requirements(task, state_diff)
print(res.score, res.reasoning, res.details)

Actual output (v0.7.1)

['ORD-1', 'ORD-1', 'ORD-1']
0 State requirements failed: missing_assertions=2 {'missing_assertions': [
  {'entity_type': 'order_items', 'record_key': 'ITEM-9106', 'field': 'item_status', 'value': 'confirmed'},
  {'entity_type': 'order_items', 'record_key': 'ITEM-9107', 'field': 'item_status', 'value': 'confirmed'}]}

ITEM-9106 and ITEM-9107 are never modified by the agent and should remain
confirmed, but they vanish from the reconstructed snapshot because all three
items collapsed under ORD-6016.

Expected output

['ITEM-1', 'ITEM-2', 'ITEM-3']
1 All required state assertions matched the saved state_diff. None

Impact

Any task asserting state on a sibling line item that is not the last in its order
gets phantom missing_assertions and a false task_completion = 0. Observed on
at least customer_support/13-cancel_partial and 15-cancel_backordered. The
same class affects orders keyed by customer_id for customers with multiple
orders (currently masked when the task modifies the order, so the diff re-adds the
correct key).

Suggested fix

Order _record_identity so a record's own id precedes owner/foreign keys.
Records without their own id (orders, customers, products) keep their previous
keying; only records that have both their own id and a parent id change:

 def _record_identity(record: dict[str, Any], fallback_index: int | None = None) -> str | None:
     for key in (
         "cart_item_id",
-        "cart_id",
-        "customer_id",
-        "order_id",
         "item_id",
         "booking_id",
         "reservation_id",
         "rental_id",
+        "cart_id",
+        "order_id",
+        "customer_id",
         "id",
         "product_id",
     ):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in state_bench/scoring.py at _record_identity, then run the supplied reproduction for customer_support/13-cancel_partial. Verify that records with their own identifiers retain distinct keys, and that evaluate_state_requirements returns the expected score with no missing assertions; check the related 15-cancel_backordered case if needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.