canonical / canonical/charm-integration-testing
wait_idle_multi_model checks each model's idleness independently, not simultaneously
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 96
Description
_Generated by an AI assistant on behalf of @rpbritton._
### Problem statement
`JubilantBackend.wait_idle_multi_model` runs an independent `wait_idle` loop per model (via `ThreadPoolExecutor`), each with its own `success_count` counter. A model is considered "done" as soon as *it individually* accumulates `count` consecutive active/idle polls, with no requirement that the other model(s) in the same call are ready at that same moment.
For CMR-style tests this can mask a still-in-progress deployment: one model can pass through a quiet window, accumulate enough consecutive successes, and return early, while the other model is still mid-operation - even though the two models are supposed to be part of one settled, integrated system.
Concrete example from a real failure: [test_execution 919474](https://test-observer.canonical.com/#/charms/409151?testExecutionId=919474&testResultId=13469759) (`mongodb-k8s` config-server/shards sharding test, run https://github.com/canonical/charm-integration-testing/actions/runs/33687828842).
Pulling the crashdump status-history for both units:
`neighbor/0` (not reported in the final error):
```
22:09:35 workload -> active
22:10:39 agent -> idle
22:16:21 workload -> maintenance ("Adding shard to config-server" - a second shard-add)
```
`neighbor` was continuously active+idle for ~5.5 minutes (22:10:39-22:16:21), comfortably long enough to satisfy its own 10-consecutive-success check and return early, before it went on to add a second shard - unrelated to the wait that had already "passed" for it.
`target/0` (the only model reported in the final error):
```
22:13:14 workload -> active ("Primary.")
22:13:16 agent -> idle
22:14:21 workload -> maintenance ("Adding shard remote-... to config-server...")
```
`target` was only active+idle for ~67 seconds before flipping back to maintenance/blocked, which was apparently too short a window to accumulate the required consecutive successful polls, so it correctly timed out after 15 minutes.
The resulting `JujuWaitTimeoutError` (and the execution metadata recorded from it) only reflects `target`'s failure. `neighbor`'s state is absent from the error even though, at the moment the error was raised, `neighbor` was *also* not in a final settled state (it had already moved on to its second shard). The per-model independence in the wait loop is what allowed `neighbor` to "pass" without the two models ever being confirmed idle *at the same time*.
### Enhancement Proposal
Change `wait_idle_multi_model` so that idleness is confirmed for all models simultaneously, not independently per model. Concretely: replace the per-model `ThreadPoolExecutor` + independent `success_count` loops with a single coordinated poll loop that, on each iteration, fetches status for all models and only counts a "success" toward the shared `success_count` when *every* model is ready in that same iteration; any model not ready in an iteration resets the shared counter. This gives a materially stronger guarantee for CMR/multi-model tests - that the whole integrated system was observed settled together - rather than each side independently passing through its own quiet window.
### What needs to get done?
- Replace the independent per-model wait loops in `JubilantBackend.wait_idle_multi_model` with a single synchronized loop that requires all models ready in the same iteration before counting a success.
- Ensure the resulting `JujuWaitTimeoutError`/`JujuWaitState` reporting still identifies which model(s) were non-compliant in the failing iteration.
- Add/update unit tests covering: all models simultaneously ready (success), one model never ready (failure, reported), and the "passes independently but not simultaneously" case this issue is based on (should now fail instead of silently succeeding for one side).
Contributor guide
Research direction
Start at the JubilantBackend.wait_idle_multi_model entry point and inspect its existing unit tests and JujuWaitTimeoutError/JujuWaitState reporting. Run the relevant unit-test suite, then verify that shared success requires every model to be ready in one poll, failures identify non-compliant models, and independently passing models no longer produce success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100