replay() returns ALLOW for a DecisionRecord whose harness was never set, with no signal it is incomplete
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 25
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
DecisionRecord.harness defaults to an empty HarnessSpan (action_type="", arguments={}) rather than None, so if application code exits the with audit(...) as d: block without ever calling d.act(...) — an early return, a caught exception on the way there, a bug in the caller — audit()'s finally clause still appends the record with that empty harness span. Nothing raises and nothing flags the record as incomplete.
replay() then hands the policy Action(type="", arguments={}, cost=0.0) unconditionally. Whether that reads as a false ALLOW depends entirely on whether the caller's policy special-cases an empty action type, which nothing in the public API asks them to do.
Minimal repro:
from auditable.chain import audit, replay, MemorySink
from auditable.record import DependencySnapshot
sink = MemorySink()
with audit("transfer_funds", snapshot=DependencySnapshot(state={"budget": 100}), sink=sink) as d:
d.read(amount=999999)
# d.act(...) never called
record = sink.records[-1]
def naive_budget_policy(state, action):
amount = action.arguments.get("amount", 0)
return amount <= state.get("budget", 0), f"amount {amount} vs budget {state.get('budget')}"
verdict = replay(record, live_state={"budget": 100}, policy=naive_budget_policy)
print(verdict.action, verdict.justified, verdict.reason)
# FixAction.ALLOW True Justified under live state.
The record is genuinely appended to the sink and genuinely replayable — it isn't rejected anywhere in the pipeline — but nothing about what the agent actually did was ever captured, and replay reports it as justified rather than as unknown.
I don't think this needs a specific fix prescribed here since it's really an API design question: possibilities that came to mind are audit() raising if the block exits without .act() having been called, a distinct verdict (e.g. INCOMPLETE) that replay returns before ever reaching a policy, or HarnessSpan tracking whether it was ever explicitly set so replay can check that instead of only checking field values. Happy to send a PR for whichever direction you'd prefer, or none at all if this is expected behaviour I'm missing context on.
Tested against a fresh clone, main branch, pytest -q green beforehand (151 passed, 21 skipped). tests/test_replay.py's three existing tests all go through a helper that calls .act(...) unconditionally, so this path doesn't appear to be covered currently.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading audit() and replay() in auditable.chain, then inspect DecisionRecord and HarnessSpan and the existing cases in tests/test_replay.py. Reproduce the no-act path from the issue and determine the intended incomplete-record behavior with the maintainer. Done means the chosen API behavior is implemented and covered by a regression test, without silently reporting the empty action as justified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100