code-yeongyu / code-yeongyu/lazycodex
ulw-loop checkpoint quality gate schema and artifact-dir errors are hard to recover from
- Dominant language
- TypeScript
- Stars
- 3.5k
- Forks
- 216
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`omo ulw-loop checkpoint --quality-gate-json` is difficult to recover from when the quality gate JSON shape or artifact location is wrong. The docs/source/runtime path give confusing signals about whether the JSON should be wrapped in a top-level `qualityGate` key, and the artifact-dir failure only becomes clear after several checkpoint attempts.
## Environment
- LazyCodex / OMO version: `4.19.4`
- Codex version: `codex-cli 0.146.0`
- OS: macOS / zsh
- Install method: Codex plugin cache under `~/.codex/plugins/cache/sisyphuslabs/omo/4.19.4`
- Target workflow: `$omo:ulw-loop` final `checkpoint --status complete` with `--quality-gate-json`
## Repository Decision
- Target repository: `code-yeongyu/lazycodex`
- Why this belongs there: the failing behavior is in the LazyCodex/OMO bundled `ulw-loop` component and skills, not upstream Codex CLI.
- LazyCodex evidence:
- Current LazyCodex source checkout: `code-yeongyu/lazycodex@fb48ddc`.
- `plugins/omo/components/ulw-loop/src/quality-gate-fields.ts:13-15` emits generic section errors like `Final quality gate is missing ${field} evidence`.
- `plugins/omo/components/ulw-loop/src/quality-gate.ts:145-151` parses the gate through `section(input, "qualityGate")`, while the workflow docs show a bare object under `--quality-gate-json`.
- `plugins/omo/skills/ulw-loop/references/full-workflow.md:197-205` documents the `--quality-gate-json` shape as top-level `codeReview`, `manualQa`, `gateReview`, `iteration`, and `criteriaCoverage` without an outer `qualityGate` wrapper.
- `plugins/omo/components/ulw-loop/src/quality-gate.ts:93-105` enforces that all referenced artifacts are inside `currentAttemptDir`.
- `plugins/omo/components/ulw-loop/src/checkpoint.ts:230-235` passes `currentAttemptDir` into quality-gate validation for evidence layout v2.
- `plugins/omo/components/ulw-loop/src/codex-goal-instruction.ts:111` documents the current-attempt artifact requirement, but users only discover the exact required directory after checkpoint rejection unless they inspect generated instructions/source.
- Upstream Codex source evidence:
- Current upstream source checkout: `openai/codex@3aae5d8`.
- `rg "ulw-loop|quality-gate-json|Final quality gate"` returned no matches in upstream Codex, so ownership is not upstream Codex CLI.
## Reproduction
This happened during a real final checkpoint close for a `rougurlike` `$omo:ulw-loop` task where all criteria were already recorded as `pass`.
1. Create/record all ULW criteria as pass for the active final story.
2. Prepare a quality gate JSON from the visible workflow docs, or wrap the same object while trying to infer the expected schema.
3. Run a checkpoint command of this form:
```bash
omo ulw-loop checkpoint \
--goal-id G001-game-studio-sprite-pipeline-users-ym \
--status complete \
--evidence "all criteria C001-C003 PASS; codeReview/manualQa/gateReview PASS" \
--codex-goal-json .omo/ulw-loop/evidence/cyberpunk-assets/final6/codex-goal-complete.json \
--quality-gate-json .omo/ulw-loop/evidence/cyberpunk-assets/final6/quality-gate.json \
--json
```
Observed failures during recovery included:
```text
ULW_LOOP_QUALITY_GATE_INVALID: Final quality gate is missing codeReview evidence
```
and then, after adjusting the JSON shape:
```text
ULW_LOOP_QUALITY_GATE_INVALID: manualQa.artifactRefs[0].path (.omo/ulw-loop/evidence/cyberpunk-assets/final6/05-battle.png) must point to an artifact from the current attempt (.omo/evidence/ulw/019fdfed-9dd0-7c30-bb65-9c6d7775c4fc/G001-game-studio-sprite-pipeline-users-ym/a0).
```
The second error is technically correct, but the workflow made it easy to build valid-looking quality-gate evidence in the wrong directory and only discover that at final checkpoint time.
## Expected Behavior
The checkpoint path should make quality-gate recovery straightforward:
1. The accepted JSON shape should be unambiguous and consistent across docs, source, and runtime.
2. If the user passes `{ "qualityGate": { ... } }` when the runtime expects a bare object, or a bare object when the runtime expects a wrapper, the error should explicitly say which form is accepted.
3. `omo ulw-loop status --json` or checkpoint preflight output should expose the exact `currentAttemptDir` that quality-gate artifact paths must live under.
4. The checkpoint error should suggest the next action, e.g. copy/move artifacts under the current attempt directory or regenerate the quality gate there.
## Actual Behavior
- A schema mismatch can surface as `Final quality gate is missing codeReview evidence`, which points at the missing nested section rather than the real issue: the top-level JSON shape is not what the runtime expected.
- The docs and source are confusing to reconcile:
- docs show a bare quality-gate object as the `--quality-gate-json` shape;
- current source parses `section(input, "qualityGate")` before reading `codeReview`.
- Artifact path validation is strict and correct, but users may only learn the exact required `currentAttemptDir` after a failed final checkpoint.
## Evidence
Runtime/environment evidence:
```text
omo --version => 4.19.4
codex --version => codex-cli 0.146.0
```
Source evidence from `code-yeongyu/lazycodex@fb48ddc`:
```text
plugins/omo/components/ulw-loop/src/quality-gate-fields.ts:13-15
section(value, field) => invalid(`Final quality gate is missing ${field} evidence.`, field)
plugins/omo/components/ulw-loop/src/quality-gate.ts:145-151
validateQualityGate(input) reads section(input, "qualityGate") before codeReview/manualQa/gateReview/iteration/criteriaCoverage.
plugins/omo/skills/ulw-loop/references/full-workflow.md:197-205
The documented --quality-gate-json example is a bare object with top-level codeReview/manualQa/gateReview/iteration/criteriaCoverage.
plugins/omo/components/ulw-loop/src/quality-gate.ts:93-105
checkFile rejects artifacts outside currentAttemptDir.
plugins/omo/components/ulw-loop/src/checkpoint.ts:230-235
checkpoint passes currentAttemptDir into validateQualityGate for evidence layout v2.
```
Upstream routing evidence:
```text
openai/codex@3aae5d8
rg "ulw-loop|quality-gate-json|Final quality gate" => no matches
```
## Root Cause
There appear to be two related UX/contract issues:
1. The `--quality-gate-json` schema contract is not consistently communicated. The source path currently suggests a top-level `qualityGate` wrapper, while the workflow docs show the bare gate object. In practice, a mismatch leads to a generic missing-section error instead of a shape-specific correction.
2. Evidence layout v2 correctly enforces `currentAttemptDir`, but that required directory is not surfaced early enough in the user-facing recovery path. Users can collect real evidence under another directory, build a complete gate, and only then discover that the artifact refs are invalid for checkpoint completion.
## Proposed Fix
- Align the docs, source, and runtime on one accepted `--quality-gate-json` shape. If backward compatibility is desired, accept both:
```ts
const gate = isRecord(input) && isRecord(input["qualityGate"])
? input["qualityGate"]
: input;
```
- Add targeted validation for wrapper mistakes:
- if the runtime expects a bare object and receives `{ qualityGate: ... }`, say: `--quality-gate-json expects the gate object directly; remove the outer qualityGate key`.
- if the runtime expects a wrapper and receives top-level `codeReview`, say: `--quality-gate-json expects { "qualityGate": ... }; wrap the gate object`.
- Include `currentAttemptDir` directly in `omo ulw-loop status --json` for the active goal, or add a `quality-gate preflight`/`checkpoint --dry-run` command that prints it.
- In the artifact path error, include a suggested recovery step such as:
```text
Copy or regenerate quality-gate artifacts under:
.omo/evidence/ulw///a/
```
- Add regression tests covering:
- wrapped vs bare quality gate JSON;
- documented sample shape;
- artifact refs outside `currentAttemptDir` with a recovery hint;
- `status --json` exposing the attempt dir used by checkpoint validation.
## Verification Plan
- Run the existing ulw-loop quality gate tests.
- Add a failing-first test using the documented quality-gate sample from `skills/ulw-loop/references/full-workflow.md`.
- Add a checkpoint validation test where artifact refs are outside the current attempt directory and assert that the error includes the actual `currentAttemptDir` plus recovery guidance.
- Run an end-to-end `omo ulw-loop checkpoint --quality-gate-json ` flow and confirm the user-facing error points directly to the schema or artifact-location issue.
---
This issue or PR was generated by LazyCodex.
Tag: lazycodex-generated
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with plugins/omo/components/ulw-loop/src/quality-gate.ts and the documented example in plugins/omo/skills/ulw-loop/references/full-workflow.md, then trace currentAttemptDir through checkpoint.ts and codex-goal-instruction.ts. Run the existing ulw-loop quality gate tests and add coverage for wrapped versus bare JSON, documented samples, artifact-location errors, and status output; done means schema and recovery guidance are consistent across source, docs, and runtime.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, developer-experience, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100