awslabs / awslabs/cli-agent-orchestrator

Two tests fail intermittently on pristine main, roughly 1 in 3 and 1 in 4, reproducible in isolation

Open
#704 4 comments 0 reactions 1 assignee Claimed by @anilkmr-a2z View on GitHub
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

## Summary

Two tests fail intermittently on pristine `upstream/main` at `5ef6d665`, with no third-party changes in the tree. Both reproduce when the single test is invoked alone, not only at full-suite scope.

They are filed together because the combined effect on a CI run is the part that matters most: if the two are independent, a full run has only about a 50% chance of hitting neither.

Observation counts are given inline so the rates can be weighed accordingly. The rates are approximate and not measured to any precision.

## Test 1

```
test/services/test_session_teardown_atomic.py::test_teardown_blocked_by_in_flight_create_same_name
```

Failure rate: roughly 1 in 3, from 7 observations.

- Three consecutive isolated runs of that single test: pass, fail, pass
- Four runs against a pristine `main` snapshot: pass, fail, fail, pass

Independent of test ordering. An earlier hypothesis that ordering was responsible was investigated and disproven -- one of the failures occurred with the suspected precondition absent.

The test appears to date from the teardown atomicity work in #498.

## Test 2

```
test/clients/test_tmux_session_exists_strict.py::TestPortableServerDetection::test_lsof_reports_the_bound_path_even_after_it_is_unlinked
```

Failure rate: roughly 1 in 4, from 10 observations.

- Five isolated runs on a branch: pass, pass, fail, pass, pass
- Five isolated runs on pristine `main`: fail, pass, pass, pass, fail

## One observation, not a diagnosis

On test 2, the failing runs took about 12-13 seconds while the passing runs took about 6. That shape is consistent with an internal timeout being reached during socket detection.

This is a correlation across 10 runs with no instrumentation, so it is offered as a starting point and not as a cause. No claim is made about which timeout, and no fix is proposed here.

Nothing comparable was observed on test 1.

## On CI

CI appears to have been green on both of these historically. Since both reproduce on unmodified `main`, that reads more like CI runs not yet having landed on the failing side than like the failures being specific to one machine. That is worth confirming from your side rather than concluding from the outside; repeated runs of the two node IDs above on a runner would settle it.

## Why it seems worth a look

At these rates a red run is close to a coin flip for a contributor, which makes a red result weak evidence that a change broke something. The practical cost is that re-running becomes the default response to a failure, and a genuine regression can pass through that habit unexamined.

Environment: Linux, tmux available, dependencies installed from the lockfile in the tree. Both tests were run alone, with coverage disabled, one invocation at a time.

Happy to collect more runs or add instrumentation if that would be useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.