stacklok / stacklok/mecatl

Investigate incoherent output from some custom Chat Completions models

Open
#1,639 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

Bug description

One model behind a custom OpenAI Chat Completions provider repeatedly returned incoherent or unrelated text through Mecatl, while another model behind the same provider and credential returned a normal answer. The affected model reportedly works through other agent harnesses against the same provider.

This was reported by a user on Discord. Across fresh attempts, a simple question produced repetition, an unrelated source-code fragment, malformed prose, and an unrelated tutorial response. Switching to a different text model from the same /models inventory produced a coherent answer; switching back reproduced the bad behavior.

This needs a product-level compatibility investigation rather than a documentation workaround. Mecatl's Chat Completions request currently sends the rendered system/conversation messages, non-strict tool schemas, stream_options.include_usage, and optional reasoning_effort. It does not expose or send temperature, top_p, or top_k, so there is no established sampling configuration users can adjust to resolve this.

Steps to reproduce

The original provider is not public in the report, so a fully hermetic reproduction still needs to be reduced:

  1. Configure a custom provider with api_flavor: openai-chat-completions.
  2. Select a compatible instruction-tuned Gemma-family model advertised by its /models endpoint.
  3. Ask a simple factual question in a fresh Mecatui session.
  4. Compare the raw request and response with the same provider/model request from a harness where it behaves normally.
  5. Repeat with another model behind the same provider to isolate provider-wide configuration from model-specific request compatibility.

Expected behavior

A model that supports the selected Chat Completions endpoint should receive a compatible request and produce coherent output. If Mecatl requires request features the model or gateway cannot support, it should reject the combination or provide actionable diagnostics rather than silently returning unusable content.

Actual behavior

The affected model returns consistently unusable content through Mecatl while a second model behind the same provider works. The current diagnostics do not expose enough sanitized request-shape information to determine whether tools, the system-message shape, streaming options, reasoning effort, a gateway chat template, or another compatibility difference is responsible.

Environment (if relevant)

  • Deployment shape: embedded Mecatui with a custom OpenAI Chat Completions provider
  • Affected model family: Gemma instruction-tuned
  • Control model family: GPT-OSS

Additional context

Reported generically from Discord; provider identity, endpoint, and credentials omitted. No secrets are included.

The investigation should first capture a sanitized wire comparison and establish whether the incoherent text is already present in the provider's streamed response or introduced during translation. Any compatibility fix should be pinned with an offline request/stream fixture.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Mecatl's Chat Completions request and streaming translation path, then capture a sanitized wire comparison for the affected and control models. Determine whether the incoherent content is present in the provider stream or introduced during translation. Done means the compatibility cause is isolated and any fix is covered by an offline request/stream fixture.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.