mpfaffenberger / mpfaffenberger/code_puppy_core_plugins
/compact fails with UnexpectedModelBehavior on GitHub Copilot models (non-streaming summarization call)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3
- Forks
- 20
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 20
Description
Description
/compact with the summarization strategy always fails when the summarization model is a GitHub Copilot model. The compaction reports success but achieves 0% reduction, so context is never actually reclaimed.
The model produces a perfectly good summary — but it's discarded during response validation and never reaches the history.
Steps to reproduce
- Use a Copilot model
- Leave summarization_model empty so it falls back to the global model.
- Fill up context, then run /compact
Actual behaviour
Compacting 132 messages using summarization strategy... (~56,502 tokens) Summarizing 18 older messages Compaction failed: [SummarizationError] LLM call failed during summarization: [UnexpectedModelBehavior] Invalid response from openai chat completions endpoint: 2 validation errors for ChatCompletion choices.0.index Input should be a valid integer [type=int_type, input_value=None, input_type=NoneType] object Input should be 'chat.completion' [type=literal_error, input_value=None, input_type=NoneType] Done! History: 132 → 132 messages via summarization Tokens: 56,502 → 56,502 (0.0% reduction)
Root cause
Three things line up:
- Copilot returns a non-spec response envelope. Captured off the wire from https://api.enterprise.githubcopilot.com/chat/completions:
top-level keys : ['choices', 'copilot_usage', 'created', 'id', 'model', 'usage']
choice[0] keys : ['finish_reason', 'message']
Missing the top-level "object": "chat.completion" discriminator and the per-choice "index". The message content itself is complete and valid.
-
The OpenAI SDK doesn't validate. It builds ChatCompletion via model_construct (openai/_models.py:1001), so both absent fields silently become None.
-
pydantic-ai validates strictly, but only on the non-streaming path. OpenAIChatModel._process_response calls _validate_completion (pydantic_ai/models/openai.py:1203), which runs a strict _ChatCompletion.model_validate(...) and raises UnexpectedModelBehavior at line 1205. None fails both int and the 'chat.completion' literal.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the /compact summarization path and inspect OpenAIChatModel._process_response and _validate_completion in pydantic_ai/models/openai.py, alongside openai/_models.py:1001. Reproduce the non-streaming GitHub Copilot response and verify that compaction accepts its valid message, updates history, and reports a reduction instead of 0%.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100