microsoft / microsoft/amplifier
Remote destinations silently lose the majority of a session's events - no self-healing replay
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.1k
- Forks
- 261
- Avg merge
- 3h 28m
- Merged PRs (30d)
- 13
Description
The problem
Against a remote destination (Azure/APIM + Entra), the telemetry hook silently fails to
deliver the large majority of a session's events — while reporting itself healthy — and the
only recovery path is a human noticing a console warning and running
context-intelligence-upload by hand.
This is not a drain-window problem (see #111 — the close_drain_timeout 10→20 bump addresses only
short tails, and says so explicitly). It is a throughput problem with no self-healing.
Measured evidence
Aggregated from the durable shutdown_undelivered records in
~/.amplifier/context-intelligence-logs/forwarding-*.jsonl on one workstation —
215 shutdowns over 15 days, 211 of them against the Azure/APIM destination:
sum median max nonzero
total 53160 163 2188 215/215
queued 31673 157 256 211/215
inflight 213 1 1 213/215
dropped 21274 0 1935 77/215
- 53,160 events never reached the server. 21,274 of those were dropped on queue overflow.
- The median session ends with 163 undelivered events and a 157-deep queue.
queuedhit thedispatch_queue_capacityceiling (256) in 4 records; 77 records shed events.breaker_open=Falseon all 215.degraded_seconds=0on 202.
That last line is the important one: the destination is healthy. Nothing is down, nothing is
rejecting, the circuit breaker never opened. The client is simply producing faster than it
delivers.
Root cause
_DestinationDispatcher runs one worker with one in-flight event at a time — the worker loop
pulls a single event, awaits _post, then pulls the next. Its delivery ceiling is therefore
roughly one event per round-trip.
A localhost POST is sub-millisecond, so this is invisible locally. A remote POST pays
connect + TLS + APIM + per-request Entra token + graph write — commonly several hundred ms. That
puts the ceiling at a few events/sec, while a busy Amplifier session emits far more. The queue
grows monotonically for the whole session, hits the 256 cap, and starts shedding.
The drain window at close() cannot help: no bounded window empties a queue that is already
150+ deep, and by then some events were dropped hours earlier.
Why this is worth fixing properly
Everything needed for silent, safe recovery already exists:
- Durable local log — every event is written to
events.jsonlbefore any forward is
attempted. Nothing is ever actually lost. - Idempotent delivery —
build_payloadstamps a deterministicidempotency_key
(SHA-256 over{event, workspace, data}) and the server dedups on it. Replaying an
already-delivered event is a no-op, so a sweep cannot create duplicates.
The missing third piece is small: a per-destination delivery watermark and a backlog sweep.
Grep the module for offset / resume / backlog and the only hits are log strings telling the
human to run the upload CLI. There is no automatic catch-up of any kind.
Suggested direction (to be designed, not prescribed here)
Sketches, in rough order of value:
- Replay on start / on reconnect. Persist a per-destination watermark; on session start (and
after the breaker closes) sweep undelivered events fromevents.jsonl, bounded by count and
age so a launch never re-POSTs a week of history. - Continuous background catch-up. Same mechanism, running during the session rather than only
at boundaries — this is what would actually keep a busy session's queue from growing all day. - Amortize the per-request cost. Batch N events per POST (needs a server-side batch endpoint)
and/or allow a small number of concurrent in-flight POSTs. Ordering and the breaker's
accounting both need thought here. - Make the warning proportional. A 2-event tail and a 2,188-event backlog currently print the
same shape of line. If the bundle recovers by itself, most of these stop needing a console
warning at all — the durableforwarding-*.jsonlrecord already preserves the trail.
Gate
This crosses a seam (client↔server boundary, networking, hook config wiring), so per AGENTS.md
"done" requires a real DTU run plus captured evidence — not unit tests alone.
Acceptance
- Given a remote destination slower than the producer, when a session ends with undelivered
events, then a subsequent session delivers them without user action and without a console
warning. - Given events already delivered, when a sweep replays them, then the server records no
duplicates (verified against a real server, not a mock). - Given a genuinely broken destination, when a sweep runs, then the failure is still surfaced
loudly — self-healing must not mask a real outage.
Generated with Amplifier
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with _DestinationDispatcher, build_payload, events.jsonl, the durable forwarding records, and AGENTS.md; grep for offset, resume, and backlog to map the current delivery path. Design and validate replay for undelivered events, duplicate suppression, and loud failure reporting, then run the required real DTU test with captured evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- api, backend, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100