microsoft / microsoft/agent-framework

.NET: Flaky SampleSmokeTest.Test_RunSample_Step5Async(InProcess_Concurrent) — "too low" vs "too high" guess mismatch

Open
#6,518 1 comment 0 reactions 1 assignee View on GitHub

@giles17 is already working on this.

Since Jun 16, 2026.

.NET
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

Summary

SampleSmokeTest.Test_RunSample_Step5Async intermittently fails in the
InProcess_Concurrent environment. The scripted playback responder expects the
workflow to emit "Your guess was too low. Try again." but the workflow emits
"Your guess was too high. Try again." The failure is non-deterministic: it
passed on re-run with no code changes, and the other two Theory variants
(InProcess_Lockstep, InProcess_OffThread) passed in the same run.

This looks like a recurrence of the same SampleSmokeTest flakiness family
previously reported and closed: #2049 (Step9 / OffThread) and #1284 (Step5 /
OffThread). The underlying non-determinism in the concurrent/off-thread
checkpointing smoke tests does not appear to be fully resolved.

Failing test

Microsoft.Agents.AI.Workflows.UnitTests.SampleSmokeTest.Test_RunSample_Step5Async(environment: InProcess_Concurrent)

Error
Assert.Equal() Failure: Values differ
Expected: Your guess was too low. Try again.
Actual:   Your guess was too high. Try again.
  at VerifyingPlaybackResponder`2.InvokeNext(TInput input) in SampleSmokeTest.cs:577
  at Step5EntryPoint.ExecuteExternalRequest(...) in Sample/05_Simple_Workflow_Checkpointing.cs:162
  at Step5EntryPoint...RunStreamToHaltOrMaxStepAsync in Sample/05_Simple_Workflow_Checkpointing.cs:125, :77
  at Step5EntryPoint.RunAsync(...) in Sample/05_Simple_Workflow_Checkpointing.cs:61
  at SampleSmokeTest.Test_RunSample_Step5Async(...) in SampleSmokeTest.cs:131
Evidence it is flaky (not a real regression)
  • Observed in CI run 27533497797 (job 81377546474); test summary: 6354 total, 1 failed.
  • Re-running the exact same commit with no changes made the test pass.
  • Only the InProcess_Concurrent Theory variant failed; InProcess_Lockstep and InProcess_OffThread passed in the same run.
  • It surfaced on an unrelated PR (a new package addition that touches no Workflows code).
Root-cause hypothesis

Test_RunSample_Step5Async drives a number-guessing-game checkpointing sample
through a VerifyingPlaybackResponder with a hard-coded sequence of
(expectedPrompt, guess) pairs across a checkpoint + restore cycle. In the
concurrent execution environment the ordering/timing of prompt emission around
the checkpoint boundary appears to diverge from the scripted sequence, so the
prompt the responder receives ("too high") does not match the expected one
("too low"). The test assumes a deterministic prompt order that the concurrent
runtime does not guarantee.

Suggested direction
  • Make the smoke test robust to prompt ordering in the concurrent environment
    (assert on game outcome rather than the exact prompt-by-prompt sequence), and/or
  • Investigate whether the concurrent checkpoint/restore path can emit a stale
    "too high/too low" prompt — i.e. whether this reflects a real ordering bug in
    the runtime rather than only a test-script assumption.
Environment
  • net10.0, ubuntu-latest, Release, integration test job

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.