yzhao062 / yzhao062/auditable

replay() returns ALLOW for a DecisionRecord whose harness was never set, with no signal it is incomplete

Open
#2 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
25
Forks
1
PR merge metrics
No merged PRs in 30d

Description

DecisionRecord.harness defaults to an empty HarnessSpan (action_type="", arguments={}) rather than None, so if application code exits the with audit(...) as d: block without ever calling d.act(...) — an early return, a caught exception on the way there, a bug in the caller — audit()'s finally clause still appends the record with that empty harness span. Nothing raises and nothing flags the record as incomplete.

replay() then hands the policy Action(type="", arguments={}, cost=0.0) unconditionally. Whether that reads as a false ALLOW depends entirely on whether the caller's policy special-cases an empty action type, which nothing in the public API asks them to do.

Minimal repro:

from auditable.chain import audit, replay, MemorySink
from auditable.record import DependencySnapshot

sink = MemorySink()
with audit("transfer_funds", snapshot=DependencySnapshot(state={"budget": 100}), sink=sink) as d:
    d.read(amount=999999)
    # d.act(...) never called

record = sink.records[-1]

def naive_budget_policy(state, action):
    amount = action.arguments.get("amount", 0)
    return amount <= state.get("budget", 0), f"amount {amount} vs budget {state.get('budget')}"

verdict = replay(record, live_state={"budget": 100}, policy=naive_budget_policy)
print(verdict.action, verdict.justified, verdict.reason)
# FixAction.ALLOW True Justified under live state.

The record is genuinely appended to the sink and genuinely replayable — it isn't rejected anywhere in the pipeline — but nothing about what the agent actually did was ever captured, and replay reports it as justified rather than as unknown.

I don't think this needs a specific fix prescribed here since it's really an API design question: possibilities that came to mind are audit() raising if the block exits without .act() having been called, a distinct verdict (e.g. INCOMPLETE) that replay returns before ever reaching a policy, or HarnessSpan tracking whether it was ever explicitly set so replay can check that instead of only checking field values. Happy to send a PR for whichever direction you'd prefer, or none at all if this is expected behaviour I'm missing context on.

Tested against a fresh clone, main branch, pytest -q green beforehand (151 passed, 21 skipped). tests/test_replay.py's three existing tests all go through a helper that calls .act(...) unconditionally, so this path doesn't appear to be covered currently.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading audit() and replay() in auditable.chain, then inspect DecisionRecord and HarnessSpan and the existing cases in tests/test_replay.py. Reproduce the no-act path from the issue and determine the intended incomplete-record behavior with the maintainer. Done means the chosen API behavior is implemented and covered by a regression test, without silently reporting the empty action as justified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.