microsoft / microsoft/amplifier

reasoning_effort is implemented by 5+ providers but absent from core's PROVIDER_CONTRACT.md

Open
#366 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.1k
Forks
261
Avg merge
3h 28m
Merged PRs (30d)
13

Description

Note: This issue is filed on microsoft/amplifier because microsoft/amplifier-core has Issues disabled. It concerns the contract gap in amplifier-core's docs/contracts/PROVIDER_CONTRACT.md.

ChatRequest.reasoning_effort (amplifier_core/message_models.py:212) is a plain str | None — no Literal, no enum, no documented vocabulary. docs/contracts/PROVIDER_CONTRACT.md contains zero mentions of it (verified by grep). The only contract-level guidance is the generic class docstring: "Providers that don't support a field ignore it."

Meanwhile at least five provider modules implement it, with four mutually incompatible vocabularies and different failure behaviors. This gap has now produced a family of real, shipped bugs.

Bugs already traced to this gap
Fix Repo What it was
#87 (merged) provider-anthropic A warning fired for an effort value that was never applied to anything
#88 (merged) provider-anthropic extended_thinking=False did not suppress output_config.effort; background chores silently ran at max reasoning effort on the primary model
#60 (open) provider-openai Explicit request.reasoning_effort was not capability-gated → hard InvalidRequestError from the live API on a non-reasoning model

Each was discovered independently. All three trace to the same root: there is no agreed contract for what this field means, what values are legal, what "off" is, or what a provider must do with a value it cannot honor.

Observed divergence

Directly verified:

Provider Accepted values Ambient config fallback Notes
anthropic low/medium/high/xhigh/max (per-model) yes falls back to instance config when request field unset
openai none/low/medium/high/xhigh yes (capability-gated) none is a real "off" token
gemini low/medium/high only no unknown values silently ignored; no off-token in this vocabulary (thinking_budget=0 is the actual off switch)
vllm openai-like partial explicit request path not capability-gated (same shape as #60)

Reported by investigation but not independently re-verified — flagged as such:

Provider Reported behavior
azure-openai field entirely unread
ollama forwarded to native think; capability-gated on both paths
github-copilot vocabulary negotiated from the model; gated on both paths; raises on malformed input
chat-completions field entirely unread
Why this is a contract question, not a provider bug

Fixing each provider individually (as the three PRs above did) does not stop the next occurrence. Concretely undecided today:

  1. Canonical vocabulary — is xhigh/max part of the portable contract, or provider-specific?
  2. The "off" value — is there a portable way to say "do not reason on this call"? Today none works on openai/copilot, is meaningless on gemini, and is unrecognized on anthropic. There is currently no portable opt-out, which is why callers reach for the non-portable extended_thinking=False kwarg — itself undocumented at contract level.
  3. Unsupported-value behavior — ignore, warn, drop, or raise? All four currently occur.
  4. Ambient-config precedence — may a provider apply its own instance-level default when the request field is unset, even when the request overrides model to something else? That specific interaction produced #88.
Suggested outcome

Specify reasoning_effort in PROVIDER_CONTRACT.md: the portable value set, portable "off" semantics, required behavior when a model cannot honor a value, and whether ambient provider config may supply a default. Providers can then be conformed to it — and the next bug in this family becomes a contract violation rather than a surprise.

Filed as an issue rather than a PR because choosing the canonical vocabulary and unsupported-value policy is a maintainer design decision, not something to assert unilaterally across eight provider repos.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/contracts/PROVIDER_CONTRACT.md and the ChatRequest definition at amplifier_core/message_models.py:212, then compare the provider behaviors and linked fixes described in the issue. Done means maintainers have decided and documented the portable vocabulary, off semantics, unsupported-value behavior, and ambient-config precedence so providers can conform to the contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.