code-yeongyu / code-yeongyu/lazycodex

lazycodex-executor-verify accepts false completion from generic receipt files

Open
#114 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
3.5k
Forks
216
PR merge metrics
No merged PRs in 30d

Description

## Summary
`$start-work` can accept false completion from `lazycodex-executor` because the shipped SubagentStop evidence gate treats any non-empty file under `.omo/evidence/` as sufficient proof. In practice this allows downstream todos to be marked complete even when the task's required deliverable is missing or stale.

## Environment
- LazyCodex version: latest `main` source synced on 2026-07-08 at `722ebec`; installed OMO plugin path observed as `.../sisyphuslabs/omo/4.16.0/...`
- Codex version: Codex Desktop thread (exact app build not exposed in this session)
- OS: macOS
- Install method: local LazyCodex/OMO plugin install
- Relevant config: orchestration through `$start-work`, Boulder state in `.omo/boulder.json`, SubagentStop hook `subagent-stop-verifying-lazycodex-executor-evidence.json`

## Repository Decision
- Target repository: `code-yeongyu/lazycodex`
- Why this belongs there: the failing behavior is implemented in LazyCodex/OMO's orchestration layer, specifically the `lazycodex-executor-verify` hook and the `start-work` / continuation contract. Upstream Codex does not ship `boulder.json`, `lazycodex-executor-verify`, or the `EVIDENCE_RECORDED` stop-hook protocol.
- LazyCodex evidence (runtime + `/var/folders/tx/mt3tgbtj0cx7xhsh1yj4ft200000gn/T/lazycodex-sources/lazycodex-source`):
- Docs and hard rules promise artifact-backed completion and distrust worker self-reports:
- `plugins/omo/components/start-work-continuation/directive.md`: "Treat every worker DoneClaim as untrusted input" and "`confirmed` is the only pass verdict"
- `plugins/omo/scripts/sync-skills.mjs`: "No completion claim while an applicable ultraqa adversarial class was never probed" and no final completion before the Global Review and Debugging Gate
- Shipped verifier only checks for `EVIDENCE_RECORDED: ` pointing to any non-empty file inside `.omo/evidence/`:
- `plugins/omo/components/lazycodex-executor-verify/src/codex-hook.ts`
- component overview at `plugins/omo/components/lazycodex-executor-verify/AGENTS.md`
- Fresh runtime repro against the shipped CLI:
1. create `.omo/evidence/receipt.txt` with placeholder text
2. omit the expected deliverable entirely (for example `.omo/notepads/opencode-facts.md`)
3. run `node plugins/omo/components/lazycodex-executor-verify/dist/cli.js hook subagent-stop` with `last_assistant_message` containing `EVIDENCE_RECORDED: .omo/evidence/receipt.txt`
4. observed result: hook stdout is empty, so the stop passes even though the deliverable is absent
- Upstream Codex source evidence from `/var/folders/tx/mt3tgbtj0cx7xhsh1yj4ft200000gn/T/lazycodex-sources/openai-codex-source`:
- no `lazycodex-executor-verify`
- no `.omo/boulder.json` orchestration layer
- no `EVIDENCE_RECORDED` stop-hook contract

## Reproduction
1. Run a `$start-work` plan whose todo requires a concrete deliverable file, for example a fact list under `.omo/notepads/...`.
2. Let a `lazycodex-executor` child stop with `last_assistant_message` ending in `EVIDENCE_RECORDED: .omo/evidence/receipt.txt`.
3. Make sure `.omo/evidence/receipt.txt` exists and is non-empty, but do not create the actual deliverable.
4. Observe that the SubagentStop verifier passes and the run can continue as if the step were evidenced.

Minimal local CLI repro used for confirmation:

```bash
mkdir -p "$TMP/.omo/evidence"
printf 'placeholder evidence\n' > "$TMP/.omo/evidence/receipt.txt"
# intentionally do NOT create the expected deliverable
node plugins/omo/components/lazycodex-executor-verify/dist/cli.js hook subagent-stop < payload.json
```

Observed on 2026-07-08:
- expected deliverable absent: `.omo/notepads/opencode-facts.md`
- hook stdout: empty

## Expected Behavior
The verifier should keep blocking unless the recorded evidence proves the task-specific deliverable exists and matches the executor's declared outcome. A generic non-empty receipt file should not be enough to close the step.

## Actual Behavior
The verifier accepts any non-empty file under `.omo/evidence/` referenced by `EVIDENCE_RECORDED: `, regardless of whether the actual deliverable exists. This creates false-positive completion and allows downstream steps to consume missing or stale outputs.

## Evidence
- Real orchestration review from a live run found:
- a todo requiring `.omo/notepads/.../opencode-facts.md` marked complete while that file did not exist
- a later todo marked complete while the expected HTML hero tagline was still unchanged
- Runtime repro on 2026-07-08 against `plugins/omo/components/lazycodex-executor-verify/dist/cli.js`:
- placeholder receipt present and non-empty
- expected deliverable absent
- CLI returned empty stdout, which is the verifier's pass condition
- Source inspection:
- `src/codex-hook.ts` only validates receipt path containment and non-empty file size
- `AGENTS.md` explicitly describes the valid receipt as any non-empty file inside `.omo/evidence/`

## Root Cause
Confirmed cause: the current `lazycodex-executor-verify` gate validates only the existence of a non-empty receipt file inside `.omo/evidence/`. It does not validate task-specific deliverables, semantic verifier output, or a `confirmed` verdict tied to the executor's declared scope. That is weaker than the documented `$start-work` contract, so false completion is possible by design in the current implementation.

## Proposed Fix
Strengthen the executor evidence contract so the hook or the next gate validates deliverable-specific evidence rather than a generic receipt:
- require a structured evidence payload that includes the expected deliverable path(s) and verifier verdict
- reject completion if the referenced deliverable is missing, empty, outside declared scope, or not mentioned by the verifier
- keep the current `.omo/evidence/` path safety checks, but make them necessary rather than sufficient
- add a regression test where `EVIDENCE_RECORDED` points to a valid non-empty receipt while the task deliverable is absent; expected result should be `decision: "block"`

Likely files/components:
- `plugins/omo/components/lazycodex-executor-verify/src/codex-hook.ts`
- `plugins/omo/components/lazycodex-executor-verify/test/codex-hook.test.ts`
- possibly the executor-side evidence format in `plugins/omo/components/ultrawork/agents/lazycodex-executor.toml`
- any `start-work` / continuation logic that currently treats receipt existence as equivalent to `confirmed`

## Verification Plan
- Reproduce the current false-positive case with a non-empty receipt and a missing deliverable; confirm it blocks after the fix
- Reproduce a valid executor stop with a real deliverable and evidence payload; confirm it passes
- Regression check that symlink/traversal/zero-byte receipt protections still hold
- Regression check that `$start-work` cannot advance a todo when the required deliverable path is absent

---
This issue or PR was generated by LazyCodex.
Tag: lazycodex-generated

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with plugins/omo/components/lazycodex-executor-verify/src/codex-hook.ts and its tests in test/codex-hook.test.ts; run the documented CLI reproduction using a non-empty receipt and missing deliverable. Trace how EVIDENCE_RECORDED becomes a pass and review the executor evidence format in lazycodex-executor.toml and start-work continuation logic. Done means the missing-deliverable regression blocks while valid evidence passes and existing path-safety checks remain covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, devtools, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.