hoangsonww / hoangsonww/Forge-Agentic-Coding-CLI
Feature: Observability Timeline with Tool Spans, Model Calls, Cost, and Validation Events
- Dominant language
- TypeScript
- Stars
- 23
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Create an observability timeline that correlates model calls, tool spans, permission decisions, validation gates, state transitions, cost, cache hits, circuit-breaker behavior, and user prompts across a Forge task.
## Problem / Opportunity
Forge has rich runtime pieces: structured logging, events, sessions, tasks, cost tracking, prompt cache, provider routing, rate limiting, circuit breakers, validation gates, and UI WebSocket streams. These are currently useful in isolation, but users need a unified task-level view to answer:
- Where did this task spend time?
- Which model/provider was called for each agent role?
- Which tool action failed or retried?
- Did validation fail because of lint, typecheck, or build?
- How much did the task cost, and what was cached?
- Did fallback routing or circuit breakers affect the outcome?
This feature would make Forge easier to debug, optimize, and trust.
## Proposed Feature
Add an observability timeline for every task:
- Structured spans for model calls, tool calls, permission prompts, validation runs, reviewer checks, and state transitions.
- CLI output via `forge task trace ` with concise and verbose formats.
- Dashboard timeline with filters for models, tools, validation, permissions, cost, and errors.
- Export support using a local JSON trace format suitable for debugging and support.
- Integration with existing logging and redaction so traces are informative but safe.
## Scope
Expected implementation areas:
- `src/logging/trace.ts`, `src/logging/logger.ts`, and `src/persistence/events.ts`.
- `src/models/router.ts`, `src/models/cost.ts`, `src/models/cache.ts`, `src/models/rate-limit.ts`, and `src/models/circuit-breaker.ts`.
- `src/agents/executor.ts`, `src/core/loop.ts`, and `src/core/validation.ts`.
- `src/tools/registry.ts` and individual tool wrappers for span boundaries.
- `src/cli/commands/task.ts` for trace output.
- `src/ui/server.ts` and UI shell timeline views.
- Unit tests for span emission, redaction, and timeline reconstruction.
## Acceptance Criteria
- [ ] Task traces include start/end timestamps, duration, status, and parent/child relationships for core spans.
- [ ] Model spans include role, provider, model, cache hit status, cost, and fallback/circuit-breaker metadata where available.
- [ ] Tool spans include tool name, risk/side effect class, permission outcome reference, duration, and success/failure.
- [ ] Validation spans distinguish typecheck, lint, build, test, or configured validation command outcomes.
- [ ] CLI trace output supports a concise human-readable view and a JSON export.
- [ ] Dashboard timeline can filter by span type and highlight failures/retries.
- [ ] Redaction is applied before traces are rendered or exported.
- [ ] Tests verify trace reconstruction from persisted events without live provider/tool calls.
## Non-Goals
- Sending traces to a hosted observability vendor by default.
- Capturing raw secrets, full prompts, or full command output in default traces.
- Replacing the existing event log or logger; this should structure and correlate them.
- Adding heavyweight telemetry dependencies without a clear need.
## Dependencies / Risks
- Span volume should be bounded so long tasks do not create excessive local storage.
- Tool output must be summarized carefully to avoid leaking credentials or overwhelming the UI.
- The trace schema should be versioned for compatibility with future replay/export tools.
- Timing measurements need to be monotonic enough for ordering even when system time changes.
## Open Questions
- Should traces be stored in the same event log or a dedicated trace file/table?
- Should verbose traces include summarized model prompts by default, or require an explicit flag?
- What retention policy should apply to trace data?
- Should trace export support Chrome trace format in addition to Forge JSON?
Contributor guide
Assessment
This issue has not been assessed yet.