canonical / canonical/data-platform-helpers

Add wait + tenacity when we restore network

Open
#20 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
1
Avg merge
14m
Merged PRs (30d)
1

Description

Hi team, just came across this issue in one of our opensearch runs:

1) test_network cut is finishing its thing
2) Juju tries to reach to the instance cut off *right before* the test restores the network:
```
2024-09-12T22:52:07.3489453Z machine-1: 22:51:43 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [35e1b8] "machine-1" cannot open api: unable to connect to API: dial tcp 10.114.131.230:17070: connect: no route to host
```

3) Network restored
4) Unit set to failure right after, because of the failed trial to run `update-status` hook:
```
2024-09-12T22:52:07.3542875Z unit-opensearch-dashboards-1: 22:52:04 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
```
5) `wait_for_idle` fails as we have a unit in error

Full CI run: https://github.com/canonical/opensearch-dashboards-operator/actions/runs/10831485374/job/30079586227

## Proposal

Let's have a wait after each `restore network cut` call with tenacity:

```
def restore_network_cut(...):
....
try_wait_model_settles(wait_list)

@tenacity.wait(...)
def try_wait_model_settles(wait_list):
# Wait for the list of units, if not specified, wait for the entire model to settle
wait_for_idle(...)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the restore_network_cut helper and the proposed try_wait_model_settles flow, then review the linked CI run to understand the race between network restoration and update-status. Add the settle wait after each restore call, with an optional wait list, and verify that wait_for_idle no longer sees the unit in error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.