microsoft / microsoft/amplifier

Remote destinations silently lose the majority of a session's events - no self-healing replay

Open
#403 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.1k
Forks
261
Avg merge
3h 28m
Merged PRs (30d)
13

Description

The problem

Against a remote destination (Azure/APIM + Entra), the telemetry hook silently fails to
deliver the large majority of a session's events
— while reporting itself healthy — and the
only recovery path is a human noticing a console warning and running
context-intelligence-upload by hand.

This is not a drain-window problem (see #111 — the close_drain_timeout 10→20 bump addresses only
short tails, and says so explicitly). It is a throughput problem with no self-healing.

Measured evidence

Aggregated from the durable shutdown_undelivered records in
~/.amplifier/context-intelligence-logs/forwarding-*.jsonl on one workstation —
215 shutdowns over 15 days, 211 of them against the Azure/APIM destination:

          sum      median    max    nonzero
total    53160      163      2188    215/215
queued   31673      157       256    211/215
inflight   213        1         1    213/215
dropped  21274        0      1935     77/215
  • 53,160 events never reached the server. 21,274 of those were dropped on queue overflow.
  • The median session ends with 163 undelivered events and a 157-deep queue.
  • queued hit the dispatch_queue_capacity ceiling (256) in 4 records; 77 records shed events.
  • breaker_open=False on all 215. degraded_seconds=0 on 202.

That last line is the important one: the destination is healthy. Nothing is down, nothing is
rejecting, the circuit breaker never opened. The client is simply producing faster than it
delivers.

Root cause

_DestinationDispatcher runs one worker with one in-flight event at a time — the worker loop
pulls a single event, awaits _post, then pulls the next. Its delivery ceiling is therefore
roughly one event per round-trip.

A localhost POST is sub-millisecond, so this is invisible locally. A remote POST pays
connect + TLS + APIM + per-request Entra token + graph write — commonly several hundred ms. That
puts the ceiling at a few events/sec, while a busy Amplifier session emits far more. The queue
grows monotonically for the whole session, hits the 256 cap, and starts shedding.

The drain window at close() cannot help: no bounded window empties a queue that is already
150+ deep, and by then some events were dropped hours earlier.

Why this is worth fixing properly

Everything needed for silent, safe recovery already exists:

  1. Durable local log — every event is written to events.jsonl before any forward is
    attempted. Nothing is ever actually lost.
  2. Idempotent deliverybuild_payload stamps a deterministic idempotency_key
    (SHA-256 over {event, workspace, data}) and the server dedups on it. Replaying an
    already-delivered event is a no-op
    , so a sweep cannot create duplicates.

The missing third piece is small: a per-destination delivery watermark and a backlog sweep.
Grep the module for offset / resume / backlog and the only hits are log strings telling the
human to run the upload CLI. There is no automatic catch-up of any kind.

Suggested direction (to be designed, not prescribed here)

Sketches, in rough order of value:

  • Replay on start / on reconnect. Persist a per-destination watermark; on session start (and
    after the breaker closes) sweep undelivered events from events.jsonl, bounded by count and
    age so a launch never re-POSTs a week of history.
  • Continuous background catch-up. Same mechanism, running during the session rather than only
    at boundaries — this is what would actually keep a busy session's queue from growing all day.
  • Amortize the per-request cost. Batch N events per POST (needs a server-side batch endpoint)
    and/or allow a small number of concurrent in-flight POSTs. Ordering and the breaker's
    accounting both need thought here.
  • Make the warning proportional. A 2-event tail and a 2,188-event backlog currently print the
    same shape of line. If the bundle recovers by itself, most of these stop needing a console
    warning at all — the durable forwarding-*.jsonl record already preserves the trail.

Gate

This crosses a seam (client↔server boundary, networking, hook config wiring), so per AGENTS.md
"done" requires a real DTU run plus captured evidence — not unit tests alone.

Acceptance

  • Given a remote destination slower than the producer, when a session ends with undelivered
    events, then a subsequent session delivers them without user action and without a console
    warning.
  • Given events already delivered, when a sweep replays them, then the server records no
    duplicates (verified against a real server, not a mock).
  • Given a genuinely broken destination, when a sweep runs, then the failure is still surfaced
    loudly — self-healing must not mask a real outage.

Generated with Amplifier

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with _DestinationDispatcher, build_payload, events.jsonl, the durable forwarding records, and AGENTS.md; grep for offset, resume, and backlog to map the current delivery path. Design and validate replay for undelivered events, duplicate suppression, and loud failure reporting, then run the required real DTU test with captured evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
api, backend, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.