microsoft / microsoft/STATE-Bench
State scorer collapses sibling `order_items` (and orders) — `_record_identity` keys child records by their parent id, causing false `task_completion = 0`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 91
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
Summary
evaluate_state_requirements reconstructs the final DB snapshot from the task-env
file, which stores entities as lists. scoring.py::_record_identity derives a
record's key from the first matching id field, but it checks owner/foreign keys
(order_id, customer_id, cart_id) before the record's own id (item_id,
…). An order_item carries both item_id and its parent order_id, so all line
items in the same order collapse under one key — only the last survives and its
siblings disappear from the reconstructed snapshot. Their state_requirements
then report as missing_assertions, failing the task even when the agent behaved
perfectly.
Environment
- STATE-Bench
v0.7.1(commita0ffc65) state_bench/scoring.py, function_record_identity(around line 206)
Reproduction
Run from a STATE-Bench checkout (uses only shipped files — task
customer_support/13-cancel_partial):
import json
from pathlib import Path
from state_bench.scoring import _record_identity, evaluate_state_requirements
from state_bench.schemas import TaskDefinition, StateDiff
# Root cause: three sibling order_items collapse to one key.
items = [{"item_id": f"ITEM-{n}", "order_id": "ORD-1", "item_status": "confirmed"} for n in (1, 2, 3)]
print([_record_identity(i) for i in items]) # -> ['ORD-1', 'ORD-1', 'ORD-1']
# End-to-end: a correct partial cancellation of only the shirt (ITEM-9105).
task = TaskDefinition.from_dict(json.loads(
Path("state_bench/domains/customer_support/tasks/13-cancel_partial.json").read_text()))
state_diff = StateDiff(modified={
"orders": {"ORD-6016": {"status": {"old": "processing", "new": "partially_cancelled"}}},
"order_items": {"ITEM-9105": {
"item_status": {"old": "confirmed", "new": "cancelled"},
"refund_amount": {"old": None, "new": 59},
"refund_method": {"old": None, "new": "credit_card"},
}},
})
res = evaluate_state_requirements(task, state_diff)
print(res.score, res.reasoning, res.details)
Actual output (v0.7.1)
['ORD-1', 'ORD-1', 'ORD-1']
0 State requirements failed: missing_assertions=2 {'missing_assertions': [
{'entity_type': 'order_items', 'record_key': 'ITEM-9106', 'field': 'item_status', 'value': 'confirmed'},
{'entity_type': 'order_items', 'record_key': 'ITEM-9107', 'field': 'item_status', 'value': 'confirmed'}]}
ITEM-9106 and ITEM-9107 are never modified by the agent and should remain
confirmed, but they vanish from the reconstructed snapshot because all three
items collapsed under ORD-6016.
Expected output
['ITEM-1', 'ITEM-2', 'ITEM-3']
1 All required state assertions matched the saved state_diff. None
Impact
Any task asserting state on a sibling line item that is not the last in its order
gets phantom missing_assertions and a false task_completion = 0. Observed on
at least customer_support/13-cancel_partial and 15-cancel_backordered. The
same class affects orders keyed by customer_id for customers with multiple
orders (currently masked when the task modifies the order, so the diff re-adds the
correct key).
Suggested fix
Order _record_identity so a record's own id precedes owner/foreign keys.
Records without their own id (orders, customers, products) keep their previous
keying; only records that have both their own id and a parent id change:
def _record_identity(record: dict[str, Any], fallback_index: int | None = None) -> str | None:
for key in (
"cart_item_id",
- "cart_id",
- "customer_id",
- "order_id",
"item_id",
"booking_id",
"reservation_id",
"rental_id",
+ "cart_id",
+ "order_id",
+ "customer_id",
"id",
"product_id",
):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in state_bench/scoring.py at _record_identity, then run the supplied reproduction for customer_support/13-cancel_partial. Verify that records with their own identifiers retain distinct keys, and that evaluate_state_requirements returns the expected score with no missing assertions; check the related 15-cancel_backordered case if needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100