canonical / canonical/charm-integration-testing

[Investigation Needed] `test_upgrade_charm` intermittently fails with `deploy incomplete`

Open
#598 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 22h
Merged PRs (30d)
96

Description

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

---

## `[injected] test_upgrade_charm` intermittently fails with `"" deploy incomplete, please try refresh again in a little bit.`

### Summary

The injected upgrade charm test intermittently fails when calling `juju refresh` to upgrade the target application. The error originates inside Juju's `refresh` command when `GetCharmURLOrigin` returns a charm origin with an empty Charmhub ID for the already-deployed application, causing Juju to report the deployment as incomplete even though the application is `active/idle`.

This has been observed across **36 test executions since March 2026** affecting at least 4 charms: `content-cache-k8s`, `loki-k8s`, `prometheus-k8s`, and `traefik-k8s`.

**This needs further investigation** to determine whether the issue is:
- A bug in Juju where bundle deployment with `revision:` override doesn't reliably store the Charmhub origin ID (Juju 3 and Juju 4 are both affected)
- A charm-integration-testing issue in how `deploy_bundle_file` is called (e.g. missing metadata that Juju needs to associate the origin)

### Observed behaviour

`juju refresh` fails immediately with:

ERROR "target" deploy incomplete, please try refresh again in a little bit.


The Juju model status at the time of failure shows the application fully healthy:

App Version Status Scale Charm Channel Rev Message
target active 1 content-cache-k8s latest/edge 111 Ready

Unit Workload Agent Address Ports Message
target/0* active idle 10.1.3.190 Ready


The command that fails:

juju refresh --model target --channel latest/edge --revision 112


The test retries the upgrade twice (as implemented in `test_upgrade_charm.py`) and fails both times with the same error.

### Steps to reproduce

This failure occurs in the `test_deploy_target_old_revision` → `test_upgrade_charm` test state sequence:

1. Neighbors are deployed (e.g. `grafana-k8s`, `mysql-k8s`, `wordpress-k8s`)
2. `test_deploy_target_old_revision` deploys the target charm at an older revision by modifying the bundle YAML with a `revision:` override and calling `deploy_bundle_file`
3. `test_upgrade_charm` calls `juju refresh --revision --channel ` to upgrade the target application

### Root cause hypothesis

The Juju `refresh` command calls `GetCharmURLOrigin` against the Juju controller to retrieve the stored Charmhub origin for the running application. If the stored origin has `Source = "charm-hub"` but `ID = ""`, Juju considers the deployment incomplete. See [`refresh.go` L395–L406](https://github.com/juju/juju/blob/main/cmd/juju/application/refresh.go).

The hypothesis is that deploying from a bundle YAML with a `revision:` override (as done in `test_deploy_target_old_revision`) may go through a code path that intermittently does not store the Charmhub origin ID in Juju's state, even though the application deploys and runs successfully.

**This needs further investigation** to determine whether the issue is:
- A bug in Juju where bundle deployment with `revision:` override doesn't reliably store the Charmhub origin ID (Juju 3 and Juju 4 are both affected)
- A charm-integration-testing issue in how `deploy_bundle_file` is called (e.g. missing metadata that Juju needs to associate the origin)

### Affected executions (sample)

| Charm | Version | Execution ID | Environment |
|---|---|---|---|
| `content-cache-k8s` | 112 | 492493| juju:3/stable ubuntu:22.04 kubernetes |
| `loki-k8s` | 227 | 521180 | juju:4/stable ubuntu:24.04 kubernetes |
| `loki-k8s` | 227 | 521130 | juju:4/stable ubuntu:24.04 kubernetes |
| `prometheus-k8s` | 293 | 522584 | juju:4/stable ubuntu:24.04 kubernetes |
| `prometheus-k8s` | 293 | 522568 | juju:4/stable ubuntu:24.04 kubernetes |
| `traefik-k8s` | 295 | 522997 | juju:3/stable ubuntu:20.04 kubernetes |

Total: 36 failures across 4+ charms since March 2026.

### Test Observer

https://test-observer.canonical.com/#/charms/407266?testExecutionId=492493&testResultId=10749157

Contributor guide

Open the contributing guide

Research direction

Start with the test_deploy_target_old_revision → test_upgrade_charm sequence in test_upgrade_charm.py and its deploy_bundle_file call. Reproduce the refresh failure, then inspect Juju's refresh.go GetCharmURLOrigin path and the stored origin for revision-overridden deployments. Done means determining whether the missing Charmhub ID comes from this test harness or Juju and identifying the appropriate fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.