aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
feat(observability): step-aware rewind semantics for the progress event stream
- Dominant language
- TypeScript
- Stars
- 143
- Forks
- 46
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 20
Description
### Component
API or orchestration, Agent (Python runtime), Scripts / CLI
### Describe the feature
Add **step-aware rewind semantics** to the append-only progress event stream so that when a durable orchestrator step or an agent pipeline phase is retried after a crash/transient failure, consumers (`bgagent watch`, `status`, notification channels) can detect the replay and **discard or supersede the now-stale events** from the abandoned attempt — instead of rendering duplicated or contradictory progress.
Today the stream is append-only and forward-cursored:
- `TaskEventsTable` is append-only; `event_id` is a ULID (time-sortable) SK (`progress_writer.py`, `ORCHESTRATOR.md`).
- `GET /v1/tasks/{id}/events` supports an `after=` forward ULID cursor (`get-task-events.ts`).
- `watch` polls forward with `afterEventId` and adaptive cadence (`cli/src/commands/watch.ts`).
Because the orchestrator is a Lambda Durable Function that **replays from the last checkpoint** after a crash (`ORCHESTRATOR.md` §durable execution), and idempotent steps may re-run, a retried step/phase can emit a *second* set of progress events for work the consumer already saw. With a pure forward cursor and no replay marker, `watch` shows both attempts: e.g. "repo_setup_complete" twice, or turn/tool events from an abandoned attempt interleaved with the live one. This is a **correctness/UX gap in the streaming surface**, not a data-loss issue (the audit log intentionally keeps everything).
### Use case
- **Trustworthy live progress:** A user watching `bgagent watch` after an orchestrator crash sees a single coherent timeline, not duplicated milestones from the replayed step.
- **Accurate notifications:** Slack/Linear/GitHub progress comments don't post contradictory "now doing X" updates from a superseded attempt.
- **Auditability preserved:** The append-only audit log still retains every event (including superseded ones); rewind is a **consumer-side presentation contract**, not a destructive delete.
### Proposed solution
**Phase 0 — Attempt/epoch stamping**
- Stamp each progress event with an **attempt/epoch identifier** tied to the durable step execution (e.g. `attempt_epoch` derived from the durable execution attempt or a monotonic per-step counter) plus the logical `step`/`phase` name. Add as optional metadata so existing readers are unaffected.
**Phase 1 — Rewind marker**
- When a step/phase begins a new attempt, write a `stream_rewind` control event carrying `{ step, superseded_after_event_id, new_epoch }`. This marks "events for `step` with an earlier epoch than `new_epoch` are superseded."
- Producers: `progress_writer.py` (runner + pipeline writers) emit the rewind marker at phase entry on retry; orchestrator emits it on durable-step retry.
**Phase 2 — Consumer honoring**
- `cli/src/commands/watch.ts` and the `status` formatter honor `stream_rewind`: when seen, drop already-rendered events for `step` with epoch < `new_epoch` (or visually mark them superseded). Forward `after` cursor semantics are unchanged.
- The `get-task-events.ts` API stays append-only; optionally add a `collapse=superseded` query mode that filters superseded events server-side for consumers that don't implement client-side rewind.
**Phase 3 — Notifications**
- Notification emitters treat a `stream_rewind` as a signal to edit/replace the prior progress message for that step rather than append a contradictory one (where the channel supports edits).
**Out of scope (explicit)**
- Deleting or mutating events in `TaskEventsTable` — the audit log remains complete and append-only.
- Changing the durable-execution retry/idempotency model itself (`ORCHESTRATOR.md`) — this issue makes its replays *legible to consumers*, it does not alter replay behavior.
### Acceptance criteria
- [ ] Progress events carry an `attempt_epoch` + logical `step`/`phase` in metadata; documented in design docs (source + Starlight sync).
- [ ] A `stream_rewind` control event is emitted on durable-step / pipeline-phase retry with `{ step, superseded_after_event_id, new_epoch }`.
- [ ] `progress_writer.py` producers emit epoch + rewind markers; covered by `agent/tests/test_progress_writer.py`.
- [ ] `bgagent watch` and `status` discard/supersede stale events on `stream_rewind`; coherent timeline verified by `cli/test`.
- [ ] `GET /v1/tasks/{id}/events` remains append-only and backward-compatible; optional `collapse=superseded` mode (if implemented) covered in `cdk/test/handlers`.
- [ ] No events are deleted from `TaskEventsTable`; audit completeness preserved (asserted by test).
- [ ] Forward `after` cursor behavior unchanged for existing consumers that ignore the new fields.
Contributor guide
Assessment
This issue has not been assessed yet.