Azure / Azure/azure-sdk-for-python

[azure-ai-agentserver-responses] Failed response inputs are replayed in conversation history

Open
#48,929 1 comment 1 reaction 0 assignees View on GitHub
Hosted Agents Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**: `azure-ai-agentserver-responses`
- **Package Version**: `2.2.0b2` on current `main`; originally observed through `agent-framework-foundry-hosting` `1.0.0b260730`
- **Operating System**: Cross-platform; reproduced with a hosted Foundry agent and locally
- **Python Version**: 3.13.14 in the original report

## Describe the bug

AgentServer persists a stored response and its `input_items` when it processes the initial `response.created` / `in_progress` event. If the handler later emits `response.failed`, `update_response()` updates the response envelope but does not change the stored input references.

`get_history_item_ids()` subsequently includes those input IDs:

- when resolving responses in a `conversation_id`; and
- when a failed standalone response is later chained through `previous_response_id`.

As a result, invalid input that caused one request to fail is replayed into subsequent otherwise-valid requests, potentially poisoning the conversation indefinitely.

Original report and reproduction: https://github.com/microsoft/agent-framework/issues/7630

Agent Framework workaround under review: https://github.com/microsoft/agent-framework/pull/7637

That workaround has to buffer synchronous handler events so it knows the terminal status before AgentServer performs its initial create, and it wraps/mutates AgentServer's private providers. It cannot address streaming and background failures cleanly without delaying their streams.

## To reproduce

1. Create a conversation.
2. Submit a response containing an unmatched `function_call_output`:

```python
response = client.responses.create(
conversation=conversation.id,
input=[
{"role": "user", "content": "Hello"},
{
"type": "function_call_output",
"call_id": "call_that_does_not_exist",
"output": "invalid output",
},
],
)
```

3. Confirm the response fails because no matching function call exists.
4. Submit a valid request to the same conversation:

```python
response2 = client.responses.create(
conversation=conversation.id,
input="Hello, how are you?",
)
```

5. Observe that the second request fails with the same unmatched-function-call error because the first request's input was included in resolved conversation history.

The equivalent issue occurs if step 2 creates a stored standalone failed response and step 4 uses `previous_response_id=response.id`.

## Actual behavior

`get_history_item_ids()` includes the failed response's own input IDs, so subsequent handlers receive and replay the invalid input.

## Expected behavior

Failed-response inputs should remain stored and retrievable through `/responses/{id}/input_items` for diagnostics, but should not be returned as replayable history for either `conversation_id` or `previous_response_id`. Successful-response history should remain unchanged.

## Ownership rationale

This behavior is defined by `azure-ai-agentserver-responses`:

- its orchestrator decides when response inputs are persisted;
- `ResponseProviderProtocol` defines create, update, and history operations;
- its in-memory and file providers build conversation and previous-response history; and
- its Foundry provider delegates to the hosted `history/item_ids` storage endpoint.

Fixing this only in Agent Framework's `foundry_hosting` adapter leaves other AgentServer hosts and streaming/background request modes with the same behavior.

## Suggested direction

Define and enforce a provider invariant that a failed response's own `input_item_ids` are excluded from replayable history while the stored items remain available for diagnostic retrieval.

- Apply this before history-limit truncation.
- Update the in-memory and file providers.
- Apply the same behavior in the Foundry `history/item_ids` backend, or expose a supported server-side filter.
- Add tests for failed conversation turns and failed standalone responses later chained with `previous_response_id`, across synchronous, streaming, and background modes.

Related:
- https://github.com/microsoft/agent-framework/issues/7630
- https://github.com/microsoft/agent-framework/pull/7637

Contributor guide

Open the contributing guide

Research direction

Start with ResponseProviderProtocol and get_history_item_ids(), then trace how the in-memory, file, and Foundry providers build conversation and previous-response history. Reproduce the failed conversation and standalone-response cases, including streaming and background modes. Done means failed inputs remain available through /responses/{id}/input_items but are excluded from replayable history, while successful history is unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
api, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.