microsoft / microsoft/agent-framework

.NET: [Flaky test] SessionFilesHostedAgentTests.UploadedFile_IsReadByHostedAgentAsync — HTTP 400 conflict despite in-test retry

Open
#5,847 0 comments 0 reactions 1 assignee View on GitHub

@rogerbarreto is already working on this.

Since May 14, 2026.

.NET
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

## Summary

`Foundry.Hosting.IntegrationTests.SessionFilesHostedAgentTests.UploadedFile_IsReadByHostedAgentAsync` is failing intermittently in the `dotnet-foundry-hosted-it` job with an HTTP 400 `conflict` from the hosted-agents service even though the test already implements a 5-attempt retry for this exact failure class.

This is **not** a flaky test in the unstable-by-default sense — it is the test catching a real eventual-consistency/identity-binding behavior in the hosted-agents service that the existing in-test mitigation can no longer absorb. Filing so it can be addressed permanently.

## Failure signature

```
System.ClientModel.ClientResultException : HTTP 400 (ServiceError: conflict)

The resource already exists or was modified concurrently. Please retry. [Request ID: 979e52b45be320cfa0e5adf2ff314cc0]

{
""code"": ""conflict"",
""message"": ""The resource already exists or was modified concurrently. Please retry. ..."",
""type"": ""invalid_request_error"",
...
}
at OpenAI.Responses.ResponsesClient.CreateResponseAsync(...)
at Azure.AI.Extensions.OpenAI.ProjectResponsesClient.CreateResponseAsync(CreateResponseOptions, CancellationToken)
at Foundry.Hosting.IntegrationTests.SessionFilesHostedAgentTests.UploadedFile_IsReadByHostedAgentAsync()
in dotnet/tests/Foundry.Hosting.IntegrationTests/SessionFilesHostedAgentTests.cs:141
```

Test wall time: `1m 02s 367ms` — consistent with all 5 retry attempts being exhausted.

## What the test already does

Step 4 of the test wraps `responses.CreateResponseAsync(readOptions)` in a bounded retry loop that catches exactly this error class:

```csharp
const int MaxAttempts = 5;
for (int attempt = 1; attempt <= MaxAttempts; attempt++)
{
try
{
rawResponse = await responses.CreateResponseAsync(readOptions);
break;
}
catch (ClientResultException ex) when (
ex.Status == 400 &&
ex.Message.Contains(""modified concurrently"", StringComparison.OrdinalIgnoreCase) &&
attempt < MaxAttempts)
{
await Task.Delay(TimeSpan.FromSeconds(2 * attempt));
}
}
```

(`dotnet/tests/Foundry.Hosting.IntegrationTests/SessionFilesHostedAgentTests.cs:135-151`)

The service is staying in conflict state for the full backoff window (2 + 4 + 6 + 8 = 20s plus call latency), so either:
1. The session/conversation revision is taking longer than 20s to settle after the `AgentSessionFiles` upload, or
2. The retry's identity is being re-bound on every attempt and never converges on the same session container.

## Suspected root cause

The test comment already calls this out:

> The platform mutates session/conversation revision when AgentSessionFiles uploads land, so an immediate /responses follow-up races and 400's with ""modified concurrently. Please retry."" — the response message literally tells us to retry. Bounded retry handles it.

Hypothesis: the platform-side write-fence after `UploadSessionFileAsync` is wider than the test's backoff envelope, or the `conversation_id` route is being recomputed by the service and momentarily disagrees with the upload's session id.

## Options to consider

1. **Stronger client-side mitigation**: increase `MaxAttempts`, lengthen backoff, or switch to exponential with jitter (e.g. up to 60s total).
2. **Service-side guarantee**: request that the hosted-agents service either (a) serialize the upload commit before acknowledging, or (b) return a retry-after hint instead of a 400 conflict.
3. **Quarantine** with `[Fact(Skip=""..."")]` short-term while #1 or #2 is implemented.

## Owner

Assigning to myself (PR #5702 area). Will handle so this stops blocking other devs.

## Related

Skip PR for the unrelated DevUI flake: #5846 / issue #5845.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.