canonical / canonical/charm-integration-testing

prodstack6 fails to provision ubuntu@20.04 and ubuntu@22.04 machines — `ubuntu:22.04 openstack` and `ubuntu:20.04 openstack` environments completely broken

Open
#684 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 22h
Merged PRs (30d)
96

Description

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

---

### Summary

The `juju:3/stable ubuntu:22.04 openstack` and `juju:3/stable ubuntu:20.04 openstack` environments on prodstack6 are unable to provision machines with ubuntu@20.04 or ubuntu@22.04 bases. Every `test_deploy` run since at least 2026-06-22 09:50 UTC has failed with the same Keystone image metadata error. The `juju:3/stable ubuntu:24.04 openstack` environment is unaffected. ubuntu@24.04 machines also provision successfully within the failing runs themselves.

### Observed Error

From Juju status at time of failure (consistent across all affected runs):

```
Machine State Address Inst id Base AZ Message
X down pending ubuntu@22.04 cannot get available image metadata: failed getting
published images metadata from keystone catalog:
cannot read inde...
```

The same error occurs for ubuntu@20.04. ubuntu@24.04 machines in the same deployments and availability zones provision without issue.

### Affected Environments

| Environment | Failures since 2026-06-22 | ubuntu@24.04 in same runs |
|---|---|---|
| `juju:3/stable ubuntu:22.04 openstack` | 27 | ✅ provisions fine |
| `juju:3/stable ubuntu:20.04 openstack` | 4 | ✅ provisions fine |
| `juju:3/stable ubuntu:24.04 openstack` | 0 | n/a |

### Scope

The issue is not specific to any charm. Affected test plans in `ubuntu:22.04 openstack` span `kubernetes-worker`, `kubernetes-control-plane`, `calico`, `canal`, `flannel`, `containerd`, `etcd`, `ceph-mon`, `ceph-osd`, `azure-integrator`, `gcp-integrator`, `easyrsa`, `manual-tls-certificates`, `lego`, and others. The `ubuntu:20.04 openstack` environment shows the same failure pattern.

Sample of affected executions (`ubuntu:22.04 openstack`):

| Execution | Test Plan | Failed bases |
|-----------|-----------|-------------|
| 332860 | kubernetes-worker / ceph-dashboard | ubuntu@22.04 ×3 |
| 332858 | kubernetes-worker / apache2 | ubuntu@20.04, ubuntu@22.04 |
| 332854 | kubernetes-worker / apache2 (multi-bundle) | ubuntu@20.04, ubuntu@22.04 ×5 |
| 332852 | kubernetes-worker / kata | ubuntu@22.04 ×2 |
| 332832 | kubernetes-worker / calico | ubuntu@22.04 |
| 332824 | gcp-integrator / kubernetes-worker | ubuntu@20.04, ubuntu@22.04 ×2 |
| 332818 | azure-integrator / kubernetes-worker | ubuntu@22.04 ×2 |
| 332808 | lego / kubernetes-worker | ubuntu@22.04 ×4 |

Sample of affected executions (`ubuntu:20.04 openstack`):

| Execution | Test Plan | Failed bases |
|-----------|-----------|-------------|
| 332811 | self-signed-certificates / kubernetes-worker | ubuntu@20.04 ×2 |
| 332809 | manual-tls-certificates / kubernetes-worker | ubuntu@20.04 ×2, ubuntu@22.04 |
| 332807 | lego / kubernetes-worker | ubuntu@20.04 ×2, ubuntu@22.04 ×2 |

### Environment

- **Cloud/Region**: `target-cloud/prodstack6`
- **Juju version**: 3.6.24
- **Confirmed affected bases**: ubuntu@20.04, ubuntu@22.04
- **Unaffected**: ubuntu@24.04 (both as a dedicated environment and within the failing runs)

### Steps to Reproduce

Run any charm integration test that deploys an application with `base: ubuntu@22.04` or `base: ubuntu@20.04` against the `ubuntu:22.04 openstack` or `ubuntu:20.04 openstack` environments on prodstack6.

### Test Observer

https://test-observer.canonical.com/#/charms/333181?testExecutionId=332828&testResultId=11866320

Contributor guide

Open the contributing guide

Research direction

Start by running any listed charm integration test against the affected prodstack6 environments and compare ubuntu@20.04 and ubuntu@22.04 provisioning with ubuntu@24.04. Investigate the repeated Keystone image metadata error using the affected execution details and Test Observer link; done means machines provision successfully for both affected bases without regressing ubuntu@24.04.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.