mpfaffenberger / mpfaffenberger/code_puppy_core_plugins

/compact fails with UnexpectedModelBehavior on GitHub Copilot models (non-streaming summarization call)

Open
#22 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3
Forks
20
Avg merge
2d 3h
Merged PRs (30d)
20

Description

Description

/compact with the summarization strategy always fails when the summarization model is a GitHub Copilot model. The compaction reports success but achieves 0% reduction, so context is never actually reclaimed.

The model produces a perfectly good summary — but it's discarded during response validation and never reaches the history.

Steps to reproduce

  1. Use a Copilot model
  2. Leave summarization_model empty so it falls back to the global model.
  3. Fill up context, then run /compact

Actual behaviour

Compacting 132 messages using summarization strategy... (~56,502 tokens) Summarizing 18 older messages Compaction failed: [SummarizationError] LLM call failed during summarization: [UnexpectedModelBehavior] Invalid response from openai chat completions endpoint: 2 validation errors for ChatCompletion choices.0.index Input should be a valid integer [type=int_type, input_value=None, input_type=NoneType] object Input should be 'chat.completion' [type=literal_error, input_value=None, input_type=NoneType] Done! History: 132 → 132 messages via summarization Tokens: 56,502 → 56,502 (0.0% reduction)

Root cause

Three things line up:

  1. Copilot returns a non-spec response envelope. Captured off the wire from https://api.enterprise.githubcopilot.com/chat/completions:

top-level keys : ['choices', 'copilot_usage', 'created', 'id', 'model', 'usage']
choice[0] keys : ['finish_reason', 'message']

Missing the top-level "object": "chat.completion" discriminator and the per-choice "index". The message content itself is complete and valid.

  1. The OpenAI SDK doesn't validate. It builds ChatCompletion via model_construct (openai/_models.py:1001), so both absent fields silently become None.

  2. pydantic-ai validates strictly, but only on the non-streaming path. OpenAIChatModel._process_response calls _validate_completion (pydantic_ai/models/openai.py:1203), which runs a strict _ChatCompletion.model_validate(...) and raises UnexpectedModelBehavior at line 1205. None fails both int and the 'chat.completion' literal.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the /compact summarization path and inspect OpenAIChatModel._process_response and _validate_completion in pydantic_ai/models/openai.py, alongside openai/_models.py:1001. Reproduce the non-streaming GitHub Copilot response and verify that compaction accepts its valid message, updates history, and reports a reduction instead of 0%.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.