microsoft / microsoft/tui-test

Session lifecycle tests flake under parallel load, and one flake reads as three

Open
#98 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
267
Forks
24
Avg merge
3d 19h
Merged PRs (30d)
50

Description

Several end-to-end tests in crates/shell-use-cli/tests/session_lifecycle.rs fail intermittently when the suite runs in parallel. Locally I see roughly one failure per three full-suite runs, each time a different test:

  • a_color_query_is_answered_over_the_pty — fixed in #95 by waiting for the report rather than for the command
  • expect_exit_code_timing_out_does_not_accept_a_stale_code
  • open_reports_the_daemon_pid_the_child_and_readiness

All pass in isolation, repeatedly. The common shape is a readiness or completion signal that is inferred rather than observed: wait command falls back to "the prompt came back and the screen is idle" when the session has no shell integration, and an idle screen is indistinguishable from a command that has not printed yet. A loaded machine widens that window.

The one already fixed shows the pattern — the probe writes nothing until it finishes, because its queries are answered by the terminal rather than echoed, so the screen was idle for the whole run. Waiting for the line the assertion reads made it deterministic. The remaining two likely want the same treatment: wait for a definite marker instead of an inferred state.

One flake looks like three

The check matrix in .github/workflows/ci.yml has no fail-fast: false, unlike the bindings matrix below it. A single flake on one platform cancels the other two, so the pull request shows three red checks and it is not obvious which one actually failed, or that only one did.

  check:
    strategy:
      fail-fast: false   # <- the bindings matrix already sets this
      matrix:

That is a one-line change and would make these much easier to read even before the tests are made deterministic.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/shell-use-cli/tests/session_lifecycle.rs, read the three named tests, and compare their waits with the fix in #95. Run the session lifecycle tests repeatedly under parallel load and identify definite output or readiness markers for the two remaining flakes. Then inspect .github/workflows/ci.yml and verify the check matrix reports independent platform failures; done means the tests stay deterministic and the matrix no longer masks failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
ci-cd, cli, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.