microsoft / microsoft/simplechat
Bound and instrument large plugin tool-result payloads in chat turns
- Dominant language
- Python
- Stars
- 152
- Forks
- 116
- Avg merge
- 7h 7m
- Merged PRs (30d)
- 122
Description
## Summary
Plugin (action) tool results are retained and passed around a chat turn without a shared size bound, and we have no diagnostics that would let us distinguish a memory problem from a Cosmos write problem from a model context-window problem.
A customer running Simple Chat locally reports the app crashing when an action returning large result sets is combined with an Azure AI Search workspace. We cannot currently diagnose that from logs, because nothing reports how much data a turn actually accumulated or where it went.
Add size instrumentation first so the failure mode can be identified, then apply a shared bounding policy for plugin results. This is not specific to any one action type.
## User Value
- Admins and developers can see why a chat turn failed instead of getting an opaque crash or a generic error.
- Actions that legitimately return large datasets degrade predictably instead of taking down the process.
- Removes the need for each action type to invent its own limits, which is the current situation and is inconsistent.
## Proposed Behavior
### Phase 1 — Diagnostics
Instrument with `debug_print` / `log_event`:
- Serialized byte size of each plugin invocation result, attributed to plugin and function name.
- Cumulative tool-result bytes accumulated for the current turn.
- Total retained size of `PluginInvocationLogger.invocations` after each append, not just the item count.
- Assembled prompt payload size immediately before the model call, broken down by conversation history, document/search citations, and tool results.
- Explicit classification of Azure OpenAI `400` responses, distinguishing `context_length_exceeded` from other bad requests.
### Phase 2 — Shared bounding policy
- A single configurable cap for plugin tool-result payloads, applied consistently rather than per action type.
- Bound what is **retained in memory**, not only what is logged or persisted.
- Surface truncation explicitly to the model and to the user, so a truncated result is never silently presented as complete.
## Acceptance Criteria
- [ ] Per-invocation and per-turn tool-result sizes appear in debug logging with plugin and function attribution.
- [ ] Retained invocation-history size is observable and bounded by bytes, not only by item count.
- [ ] Prompt assembly logs a size breakdown before the model call.
- [ ] Azure OpenAI context-length errors are logged distinctly and surfaced as an actionable message rather than a generic failure.
- [ ] A shared plugin result cap exists and applies regardless of action type.
- [ ] Truncation is explicitly marked in the result handed to the model.
- [ ] Functional test covering an oversized plugin result, asserting bounding and truncation signaling.
## Notes
### Current state
- `PluginInvocationLogger.__init__` (`semantic_kernel_plugins/plugin_invocation_logger.py` L343) retains `max_history = 1000` invocations holding the full, untruncated `result` object. Truncation via `sanitize_plugin_invocation_value` (`max_string_length=500`) and `MAX_SAFE_INVOCATION_STRING_LENGTH = 20000` is applied to logging and persistence output only, never to retained state.
- `get_invocations_for_conversation(..., limit=1000)` is called from at least eight sites in `route_backend_chats.py` during a single turn, including L3000, L3024, L18511, L18874, and L19418.
- Per-action-type limits exist and are inconsistent: `MCP_MAX_TOOL_RESULT_TEXT_LENGTH = 120000` in `functions_mcp_operations.py`; the Yamcs action defaults to `max_rows = 500` and `byte_limit = 250000`; several other action types have no cap at all.
- Search contributes independently: `SEARCH_DEFAULT_TOP_N = 12` and `SEARCH_MAX_TOP_N = 500` in `functions_search.py`.
- `conversation_history_limit` (default 10) replays prior messages, which may already carry large tool payloads from earlier turns.
### Pattern to reuse
`build_evidence_envelope` in `functions_mixed_source_orchestration.py` L1121 already implements bounded JSON lists and UTF-8-safe truncation with an explicit `bounds_applied` flag. That is the shape the shared policy should follow rather than inventing a new one.
### Leading hypothesis
Process memory exhaustion from retained full-size invocation results, not model context overflow. A context overflow would return a clean Azure OpenAI `400`, which the customer is not seeing. Retaining up to 1000 full results and re-reading them from many call sites per turn is the mechanism that scales badly. This is masked in App Service by gunicorn worker recycling and available RAM, and manifests as a hard process exit on a developer workstation running other workloads alongside it.
Secondary candidates to confirm or eliminate with the same instrumentation, rather than assume:
- Cosmos DB 2 MB item size limit on message save when citations carry large tool results. The `log_tool_execution` path in `agent_logging_chat_completion.py` L39 truncates `function_result` to 500 characters, but citations built from the invocation logger take a different path that needs verifying.
- Repeated deep copy and serialization of large results across the sanitize → citation → persist chain.
- The Werkzeug development server with `threaded=True` being less resilient under memory pressure than gunicorn.
### Related gap
`route_backend_chats.py` L14291 is the only place that pattern-matches `BadRequestError`, and it does not distinguish context-length failures. If the model is reporting an oversized prompt, we are currently discarding that signal.
### Staging
Phase 1 is independently valuable and should ship first. Phase 2 should not be scoped until Phase 1 data identifies the actual failure mode.
Contributor guide
Assessment
This issue has not been assessed yet.