canonical / canonical/notebook-operators
Test times out waiting for model
- Dominant language
- Python
- Stars
- 6
- Forks
- 12
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 12
Description
### Bug Description
This looks to be the same issue as https://github.com/canonical/resource-dispatcher/issues/86
While all the integration tests are passing though, only the bundle tests are the ones that fail
1. [workflow yaml](https://github.com/canonical/notebook-operators/blob/c56a97fabadaa19f8255dbd205a75cf12bf55f17/.github/workflows/ci.yaml#L185-L193)
2. [test_charms.py](https://github.com/canonical/notebook-operators/blob/c56a97fabadaa19f8255dbd205a75cf12bf55f17/tests/test_charms.py)
Locally though I haven't managed to reproduce as I'm bumping into a different error that I'll share in the comments.
https://github.com/canonical/notebook-operators/actions/runs/13542756086/job/37848778215?pr=435
### To Reproduce
1. Trigger the `on_pull` event from commit c56a97fabadaa19f8255dbd205a75cf12bf55f17
### Environment
1. Code from commit c56a97fabadaa19f8255dbd205a75cf12bf55f17
2. [MicroK8s, JuJu](https://github.com/canonical/notebook-operators/blob/c56a97fabadaa19f8255dbd205a75cf12bf55f17/.github/workflows/ci.yaml#L169-L175)
### Relevant Log Output
```shell
tests/test_charms.py::test_prometheus_grafana_integration
-------------------------------- live log call ---------------------------------
INFO juju.model:model.py:2284 Deploying ch:amd64/focal/prometheus-k8s-228
...
INFO juju.model:model.py:3211 Waiting for model:
grafana-k8s/0 [idle] maintenance: installing charm software
INFO juju.model:model.py:3211 Waiting for model:
grafana-k8s/0 [idle] maintenance: installing charm software
FAILED
------------------------------ live log teardown -------------------------------
INFO httpx:_client.py:1038 HTTP Request: DELETE https://10.1.0.18:16443/api/v1/namespaces/kubeflow-user "HTTP/1.1 200 OK"
INFO httpx:_client.py:1038 HTTP Request: DELETE https://10.1.0.18:16443/api/v1/namespaces/kubeflow-user/serviceaccounts/default-editor "HTTP/1.1 200 OK"
INFO pytest_operator.plugin:plugin.py:903 Model status:
Model Controller Cloud/Region Version SLA Timestamp
kubeflow github-pr-d0cb9-microk8s microk8s/localhost 3.6.2 unsupported 12:20:21Z
App Version Status Scale Charm Channel Rev Address Exposed Message
admission-webhook active 1 admission-webhook latest/edge 404 10.152.183.83 no
grafana-k8s 9.5.3 maintenance 1 grafana-k8s latest/stable 133 10.152.183.92 no installing charm software
istio-ingressgateway active 1 istio-gateway latest/edge 1378 10.152.183.129 no
istio-pilot active 1 istio-pilot latest/edge 1321 10.152.183.193 no
jupyter-controller active 1 jupyter-controller 0 10.152.183.164 no
jupyter-ui active 1 jupyter-ui 0 10.152.183.24 no
kubeflow-dashboard active 1 kubeflow-dashboard latest/edge 747 10.152.183.123 no
kubeflow-profiles active 1 kubeflow-profiles latest/edge 539 10.152.183.170 no
prometheus-k8s 2.52.0 active 1 prometheus-k8s latest/stable 228 10.152.183.101 no
prometheus-scrape-config-k8s n/a active 1 prometheus-scrape-config-k8s latest/stable 56 10.152.183.191 no
Unit Workload Agent Address Ports Message
admission-webhook/0* active idle 10.1.72.80
grafana-k8s/0* maintenance idle 10.1.72.89 installing charm software
istio-ingressgateway/0* active idle 10.1.72.75
istio-pilot/0* active idle 10.1.72.74
jupyter-controller/0* active idle 10.1.72.79
jupyter-ui/0* active idle 10.1.72.78
kubeflow-dashboard/0* active idle 10.1.72.82
kubeflow-profiles/0* active idle 10.1.72.81
prometheus-k8s/0* active idle 10.1.72.88
prometheus-scrape-config-k8s/0* active idle 10.1.72.85
...
=================================== FAILURES ===================================
_____________________ test_prometheus_grafana_integration ______________________
Traceback (most recent call last):
...
File "/home/runner/work/notebook-operators/notebook-operators/tests/test_charms.py", line 347, in test_prometheus_grafana_integration
await ops_test.model.wait_for_idle(status="active", timeout=60 * 20)
File "/home/runner/work/notebook-operators/notebook-operators/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 3209, in wait_for_idle
raise jasyncio.TimeoutError("Timed out waiting for model:\n" + busy)
asyncio.exceptions.TimeoutError: Timed out waiting for model:
grafana-k8s/0 [idle] maintenance: installing charm software
```
### Additional Context
I can't manage to reproduce this error when running the tests locally though.
Contributor guide
Research direction
Start with .github/workflows/ci.yaml lines 185-193 and tests/test_charms.py, especially test_prometheus_grafana_integration, using commit c56a97fabadaa19f8255dbd205a75cf12bf55f17. Run the bundle test and inspect the Grafana unit remaining in maintenance while wait_for_idle times out. Done when the test completes reliably in the pull-request workflow without this timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, kubernetes, python
- Domain
- ci-cd, infrastructure, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100