canonical / canonical/data-platform-helpers
Add wait + tenacity when we restore network
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 14m
- Merged PRs (30d)
- 1
Description
Hi team, just came across this issue in one of our opensearch runs:
1) test_network cut is finishing its thing
2) Juju tries to reach to the instance cut off *right before* the test restores the network:
```
2024-09-12T22:52:07.3489453Z machine-1: 22:51:43 [91mERROR[0m juju.worker.dependency "api-caller" manifold worker returned unexpected error: [35e1b8] "machine-1" cannot open api: unable to connect to API: dial tcp 10.114.131.230:17070: connect: no route to host
```
3) Network restored
4) Unit set to failure right after, because of the failed trial to run `update-status` hook:
```
2024-09-12T22:52:07.3542875Z unit-opensearch-dashboards-1: 22:52:04 [94mINFO[0m juju.worker.uniter awaiting error resolution for "update-status" hook
```
5) `wait_for_idle` fails as we have a unit in error
Full CI run: https://github.com/canonical/opensearch-dashboards-operator/actions/runs/10831485374/job/30079586227
## Proposal
Let's have a wait after each `restore network cut` call with tenacity:
```
def restore_network_cut(...):
....
try_wait_model_settles(wait_list)
@tenacity.wait(...)
def try_wait_model_settles(wait_list):
# Wait for the list of units, if not specified, wait for the entire model to settle
wait_for_idle(...)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the restore_network_cut helper and the proposed try_wait_model_settles flow, then review the linked CI run to understand the race between network restoration and update-status. Add the settle wait after each restore call, with an optional wait list, and verify that wait_for_idle no longer sees the unit in error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100