monad-developers / monad-developers/ultrafuzz
report: populate run metrics and remove obsolete strategy-loop/digest sections
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 18
- Avg merge
- 11h 10m
- Merged PRs (30d)
- 194
Description
Problem
Generated report.md and run-status projections have several generic reporting-quality problems:
- Elapsed time, models used, tokens used, and estimated spend can appear as
unavailableeven when durable run/session accounting evidence permits attribution. - Multi-message, retry, and interrupted CLI activity can be undercounted when usage is retained only at terminal invocation settlement.
- Multiple agent invocations inside one workflow attempt, such as schema-correction calls, can overwrite rather than accumulate usage.
- A persisted usage snapshot and its correlated accounting event can diverge if they are committed independently.
- Provider-inclusive input/output counters, cumulative snapshots, inherited cache ratios, and adapter-recorded cost can be projected inconsistently across runtime, CLI, eval, and worker consumers.
- A detached continuation can be admitted as
runningand later finish or fail while mutable localstate.jsonremains live until a successful reconciliation command projects the linked workflow terminal state. - The
Strategy loopssection duplicates audit-profile topology and is no longer useful now that the audit profile is the canonical orchestration layer. - Low-level digest material is implementation/provenance noise in the developer-facing report.
- Strategy-loop attribution can be misleading when a deduplicated finding was observed by only a subset of executions.
Expected behavior
- Populate elapsed time, model usage, token usage, and estimated spend from durable accounting evidence, with explicit unavailable/partial semantics only when evidence is genuinely absent.
- Preserve adapter-recorded event cost for the events it covers, distinguish provider-returned charges from client-catalogue estimates by documented provenance, and never present an estimate as authoritative provider billing or complete spend when coverage is partial.
- Count every billable response exactly once across progress, retries, handled interruption, multiple invocations in one attempt, cumulative snapshots, and downstream projections.
- Commit each owned usage snapshot and its exact correlated accounting event atomically.
- Detect local-live/linked-workflow-terminal divergence and avoid presenting stale local
runningstate as the current linked workflow outcome. - Remove obsolete strategy-loop and digest presentation from
report.mdwhile retaining validated structured provenance. - Represent attempts, variants, and finding observations accurately;
2/3-style evidence must map to actual contributing executions and not be confused with deduplicated finding counts. - Add contract and regression coverage for complete/partial/unavailable metadata, cumulative usage, interrupted processes, lifecycle divergence, section removal, and multi-execution attribution.
Scope
This is a generic reporting/accounting issue. It must not depend on or disclose any audited project, private target, or target-specific finding.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing report.md generation, durable usage accounting, and state.json reconciliation across runtime, CLI, eval, and worker consumers. Review the existing contract and regression coverage, then verify complete, partial, and unavailable metadata, atomic cumulative accounting, lifecycle divergence, removed report sections, and accurate multi-execution attribution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, cli, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100