canonical / canonical/charm-integration-testing
CMR test_deploy timeout: neighbor unit stuck in waiting, target controller-service is ClusterIP-only (no LoadBalancer)
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 96
Description
**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa
### Summary
Cross-model-relation (CMR) integration test runs in the `juju:3/stable ubuntu:24.04 cmr/kubernetes/openstack` environment (target model on a `kubernetes` cloud, neighbor model on an `openstack` cloud) consistently time out waiting for the neighbor-side unit to reach `active`. A live reproduction found the target controller's `controller-service` Kubernetes Service is provisioned as `ClusterIP` only, with no `LoadBalancer`/external IP.
### Investigation finding
Using workflow run `https://github.com/canonical/charm-integration-testing/actions/runs/34454942724`, which reproduced the identical symptom (neighbor unit stuck `waiting`, timeout) on the same shared environment config as the test executions below, we used its debug pause to inspect the live controller:
```
$ kubectl describe svc/controller-service -n controller-charmqa-34454942724-
...
Type: ClusterIP
IP:
...
```
No `LoadBalancer`/external IP was present. Juju's bootstrap config exposes `controller-service-type` (`cluster`/`loadbalancer`/`external`) plus `controller-external-name`/`controller-external-ips` to control this; `static/environments.yaml` does not currently set any of these for the k8s cloud pool used here, so it defaults to `cluster`. A `ClusterIP`-only Service is only reachable from within its own cluster's pod network.
**Note:** Juju's archived logs for a test execution don't capture k8s Service definitions, so this couldn't be shown from 668730/572732's own log archives alone — it was confirmed via the live, identically-failing reproduction above.
### Affected charms / revisions observed
- `vault-k8s` rev 565 (track `2.0`, risk `stable`) as target, `self-signed-certificates` rev 586 (track `1`, risk `stable`) co-deployed on target, `vault` rev 710 (track `2.0`, risk `stable`) as neighbor
- `argo-controller` rev 939 (track `3.7`, risk `edge`) as target, `self-signed-certificates` rev 586 and `minio` rev 583 co-deployed on target, `opentelemetry-collector` rev 315 as neighbor
### Failing test
`charm_integration_testing.test_suite.deploy.test_deploy`
### Observed error (verbatim)
Test execution 668730 (`vault-k8s`):
```
juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active, blocked], units: [active, blocked], unit agents: [idle] (applications: ['neighbor'], units: ['neighbor/0'])
```
Charm status at failure: `application:waiting:Waiting for CA certificate in workload`, `unit:waiting:Waiting for CA certificate in workload`
Test execution 572732 (`argo-controller`):
```
juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active], units: [active] (applications: ['neighbor'], units: ['neighbor/0'])
```
### Environment / steps to reproduce
- Target model: `self-signed-certificates` + `vault-k8s`, related via `tls-certificates-pki`, offer `neighbor-offer` exposing `vault-pki`, bootstrapped on a `kubernetes` cloud.
- Neighbor model: `vault`, consuming the offer via SAAS, related via `tls-certificates-access`, bootstrapped on an `openstack` cloud.
- Deploy both bundles, relate target↔neighbor via the SAAS offer, wait for all applications/units to reach `active`.
- Result: target-side units settle; neighbor-side unit never leaves `waiting`/times out.
### Additional observation: LoadBalancer provisioning conflict (unconfirmed if still current)
While testing a candidate `controller-service-type=loadbalancer` bootstrap config against the same `kubernetes` cloud/region (temporary controller, not part of the original test executions above), we hit:
```
Error syncing load balancer: failed to ensure load balancer: error creating loadbalancer
kube_service_k8s-__controller-service:
error creating loadbalancer {...,"vip_subnet_id":"","provider":"amphora",...}:
Expected HTTP response code [201 202] when accessing [POST https:///v2.0/lbaas/loadbalancers],
but got 500 instead: {"faultcode": "Server", "faultstring": "Provider 'amphora' reports error: ConflictException: 409", "debuginfo": null}
```
This recurred across two attempts that reused the same controller name, so it may be a naming collision from our own cleanup rather than a persistent platform issue — we have not been able to re-verify whether this still occurs, since the interactive debug session used for this investigation has since expired. Related reports on this infrastructure: #52, #1004.
### Test Observer links
- https://test-observer.canonical.com/#/charms/409648?testExecutionId=668730&testResultId=13650874
- https://test-observer.canonical.com/#/charms/408555?testExecutionId=572732&testResultId=12101045
### Note on triage classification
This failure pattern (timeout waiting for units) would normally be classified as transient/environmental and not warrant a filed issue. Filing here because it recurred across at least 8 test executions over several hours without resolving, identically across two unrelated charms, and a live reproduction points to a concrete, non-transient mechanism (ClusterIP-only controller Service) rather than flakiness.
Contributor guide
Research direction
Start with charm_integration_testing.test_suite.deploy.test_deploy and static/environments.yaml, then inspect the Kubernetes controller-service configuration used by the CMR environment. Reproduce the cross-cloud deployment and verify whether the controller is reachable from the neighbor model. Done means the neighbor unit reaches active without the recurring timeout, with any LoadBalancer provisioning conflict understood or documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- infrastructure, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100