canonical / canonical/charm-integration-testing
[Investigation Needed] `test_upgrade_charm` intermittently fails with `deploy incomplete`
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 96
Description
**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa
---
## `[injected] test_upgrade_charm` intermittently fails with `"" deploy incomplete, please try refresh again in a little bit.`
### Summary
The injected upgrade charm test intermittently fails when calling `juju refresh` to upgrade the target application. The error originates inside Juju's `refresh` command when `GetCharmURLOrigin` returns a charm origin with an empty Charmhub ID for the already-deployed application, causing Juju to report the deployment as incomplete even though the application is `active/idle`.
This has been observed across **36 test executions since March 2026** affecting at least 4 charms: `content-cache-k8s`, `loki-k8s`, `prometheus-k8s`, and `traefik-k8s`.
**This needs further investigation** to determine whether the issue is:
- A bug in Juju where bundle deployment with `revision:` override doesn't reliably store the Charmhub origin ID (Juju 3 and Juju 4 are both affected)
- A charm-integration-testing issue in how `deploy_bundle_file` is called (e.g. missing metadata that Juju needs to associate the origin)
### Observed behaviour
`juju refresh` fails immediately with:
ERROR "target" deploy incomplete, please try refresh again in a little bit.
The Juju model status at the time of failure shows the application fully healthy:
App Version Status Scale Charm Channel Rev Message
target active 1 content-cache-k8s latest/edge 111 Ready
Unit Workload Agent Address Ports Message
target/0* active idle 10.1.3.190 Ready
The command that fails:
juju refresh --model target --channel latest/edge --revision 112
The test retries the upgrade twice (as implemented in `test_upgrade_charm.py`) and fails both times with the same error.
### Steps to reproduce
This failure occurs in the `test_deploy_target_old_revision` → `test_upgrade_charm` test state sequence:
1. Neighbors are deployed (e.g. `grafana-k8s`, `mysql-k8s`, `wordpress-k8s`)
2. `test_deploy_target_old_revision` deploys the target charm at an older revision by modifying the bundle YAML with a `revision:` override and calling `deploy_bundle_file`
3. `test_upgrade_charm` calls `juju refresh --revision --channel ` to upgrade the target application
### Root cause hypothesis
The Juju `refresh` command calls `GetCharmURLOrigin` against the Juju controller to retrieve the stored Charmhub origin for the running application. If the stored origin has `Source = "charm-hub"` but `ID = ""`, Juju considers the deployment incomplete. See [`refresh.go` L395–L406](https://github.com/juju/juju/blob/main/cmd/juju/application/refresh.go).
The hypothesis is that deploying from a bundle YAML with a `revision:` override (as done in `test_deploy_target_old_revision`) may go through a code path that intermittently does not store the Charmhub origin ID in Juju's state, even though the application deploys and runs successfully.
**This needs further investigation** to determine whether the issue is:
- A bug in Juju where bundle deployment with `revision:` override doesn't reliably store the Charmhub origin ID (Juju 3 and Juju 4 are both affected)
- A charm-integration-testing issue in how `deploy_bundle_file` is called (e.g. missing metadata that Juju needs to associate the origin)
### Affected executions (sample)
| Charm | Version | Execution ID | Environment |
|---|---|---|---|
| `content-cache-k8s` | 112 | 492493| juju:3/stable ubuntu:22.04 kubernetes |
| `loki-k8s` | 227 | 521180 | juju:4/stable ubuntu:24.04 kubernetes |
| `loki-k8s` | 227 | 521130 | juju:4/stable ubuntu:24.04 kubernetes |
| `prometheus-k8s` | 293 | 522584 | juju:4/stable ubuntu:24.04 kubernetes |
| `prometheus-k8s` | 293 | 522568 | juju:4/stable ubuntu:24.04 kubernetes |
| `traefik-k8s` | 295 | 522997 | juju:3/stable ubuntu:20.04 kubernetes |
Total: 36 failures across 4+ charms since March 2026.
### Test Observer
https://test-observer.canonical.com/#/charms/407266?testExecutionId=492493&testResultId=10749157
Contributor guide
Research direction
Start with the test_deploy_target_old_revision → test_upgrade_charm sequence in test_upgrade_charm.py and its deploy_bundle_file call. Reproduce the refresh failure, then inspect Juju's refresh.go GetCharmURLOrigin path and the stored origin for revision-overridden deployments. Done means determining whether the missing Charmhub ID comes from this test harness or Juju and identifying the appropriate fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100