stacklok / stacklok/mecatl

diagnostics: retain bounded structural summaries for successful and incomplete model streams

Open
#1,676 0 comments 0 reactions 1 assignee View on GitHub

@jhrozek is already working on this.

Since Sep 18, 2026.

Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

Summary

Mecatl retains content-free request manifests and sanitized evidence for failed or suppressed provider attempts. It deliberately retains no structural evidence for a successful HTTP/model stream.

That leaves a gap for reports such as #1639:

  • the provider returns HTTP 200;
  • the stream appears to finish;
  • the normalized assistant text is nonsensical; and
  • there is no durable evidence of whether Mecatl saw normal framing, a provider terminal event, an incomplete stream, or parser/decoder anomalies.

Add target-bound, durable, content-free structural stream evidence. Do not capture raw provider traffic.

Tracking: #1639

This issue improves future diagnosis of reports like #1639. It does not close, resolve, or claim to reproduce #1639.

Current behavior

network.attempt records failed or suppressed provider attempts. ADR 0255 explicitly excludes successful-attempt timing and protocol-phase evidence.

For a clean HTTP 200 stream that produces anomalous text, the existing failed-attempt path may contain no evidence at all.

Proposed behavior

For each provider attempt, retain a bounded structural summary that can answer questions such as:

HTTP status: 200
response framing: SSE
provider terminal frame observed: yes
protocol events accepted: 73
response bytes read: 14,220
local stream outcome: complete

The exact field names and closed vocabularies belong in the acceptance plan and decision record. The design must support at least these outcomes:

  • complete;
  • incomplete;
  • stream error;
  • cancelled; and
  • evidence unavailable.

The debugger must describe these as observations, not causal conclusions.

For example:

Mecatl observed SSE framing and a provider terminal frame.

is permitted.

The gateway caused the bad response.

is not a conclusion this evidence can support on its own.

Required architecture constraints

  • Provider adapters observe provider-specific stream structure.
  • A provider-neutral, run-local observer carries only validated structural facts to the agent loop.
  • The agent loop remains the sole event producer.
  • The server relay remains the sole durable EventLog writer.
  • Evidence is target-bound and available only through the session debugger.
  • Ordinary gRPC, HTTP/SSE, Team, and event-log-readback streams do not expose it.
  • Retried provider attempts remain separately correlated to the same logical turn.
  • Missing support is explicit as unavailable; it is never presented as a healthy stream.
  • The implementation must preserve streaming behavior: no ReadAll, no response buffering before delivery, and no change to TTFT, retry, cancellation, or stream-idle semantics.

Strict privacy boundary

This issue must never retain or expose:

  • request, prompt, message, tool-schema, argument, or tool-result contents;
  • response text or response fragments;
  • raw errors or provider error codes;
  • URLs, query strings, headers, cookies, tokens, credentials, or environment values;
  • provider request/correlation IDs; or
  • response-content hashes that become an offline content oracle.

This issue must not add --capture-wire, a capture directory, a raw-response debug mode, or a plaintext diagnostic artifact.

Why this is separate from request-manifest evidence

The request manifest answers:

What content-free request shape did Mecatl prepare?

This issue answers:

What content-free stream structure did Mecatl observe afterward?

A normal manifest cannot show whether the gateway ended an SSE stream correctly. A normal stream summary cannot show how many tool-schema bytes were advertised. The two records diagnose different parts of one turn.

Decision record

Architectural. ADR 0255 explicitly says successful attempts and protocol-phase timings are not measured. This issue requires a new or superseding decision record that defines the new retained-evidence boundary, its vocabulary, validation, visibility, and retention semantics.

Acceptance criteria

  • A completed provider attempt produces one bounded, validated structural summary.
  • A truncated or errored stream records incomplete/error structure without fabricating a normal terminal frame.
  • Retry attempts remain distinguishable and correlate to their logical turn.
  • Invalid provider-produced evidence is rejected and never persisted.
  • All supported in-tree production provider families have offline fixture coverage.
  • Durable relay persistence survives a disconnected client without making a diagnostic persistence failure fail inference.
  • Ordinary client streams and public event-log readback omit this evidence.
  • The target-bound debugger exposes bounded summaries and explicitly reports unavailable, incomplete, and truncated evidence.
  • Debugger wording reports observations only and does not assign fault to Mecatl, a gateway, or a model.
  • Engine API compatibility artifacts, changelog, documentation, and the superseding decision record are updated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.