[Investigate] `test_deploy_target_old_revision` times out for `sloth-k8s` rev 12 on Juju 4.0.5

**Note:** This issue was generated with AI assistance (GitHub Copilot) on behalf of @canonical/solutions-qa . The log excerpts and observations below are taken directly from Test Observer and Juju debug logs. Further human investigation is recommended before drawing conclusions.

### Summary

The test `test_deploy_target_old_revision` consistently times out when freshly deploying `sloth-k8s` revision 12 (`latest/edge`) on Juju 4.0.5. After the Juju provisioner creates all required Kubernetes resources and declares the application ready, the application remains at status `unknown 0/1` for the entire 15-minute wait window. The Juju unit agent (`unit-target-0`) produces no log output whatsoever during this period. The failure is 100% reproducible across 5 test executions on the `juju:4/stable ubuntu:24.04 kubernetes` environment. Both `test_deploy` (fresh deploy of rev 19) and `test_downgrade_charm` (in-place `juju refresh` to rev 12) pass in the same execution on the same Juju version.

---

### Environment

| Field | Value |
|---|---|
| Charm under test | `sloth-k8s` rev 19, `latest/edge` |
| Revision being deployed by failing test | `sloth-k8s` rev 12, `latest/edge` |
| Test plan | `integration/loki-k8s:logging/loki_push_api/sloth-k8s:logging` |
| Environment | `juju:4/stable ubuntu:24.04 kubernetes` |
| Juju version | 4.0.5 |
| Co-deployed charm | `loki-k8s` rev 217, `2/stable` |

---

### Failing test

**`test_deploy_target_old_revision`**

This test selects a previously-passing historical revision of the charm under test (rev 12, validated in an earlier execution where `test_deploy` passed) and freshly deploys it via a generated bundle. It then waits up to 15 minutes for all applications to reach `active`.

---

### Observed behaviour

#### Error (verbatim from `io_log`)

```
juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active], units: [active] (applications: ['target'])
```

#### `juju status` at time of log collection

```
target sloth-k8s latest/edge 12 unknown 0/1
```

The application has `scale: 1` in model state but `0/1` units active. No unit agent status is present; `juju-status` shows no `units:` section under `target`.

#### Juju provisioner behaviour (from `debug-log.txt`)

The `caasapplicationprovisioner` worker for the fresh-deploy application creates all Kubernetes resources and completes its initial loop:

```
controller-0: 2026-05-26 01:16:47 INFO created CAAS application "target" with ID "abb89aaa-b068-464e-8333-c02e398dad28"
controller-0: 2026-05-26 01:16:51 DEBUG ensuring application "target" exists
controller-0: 2026-05-26 01:16:53 DEBUG application "target" was "updated"
controller-0: 2026-05-26 01:16:53 DEBUG application "target" is ready
controller-0: 2026-05-26 01:16:53 DEBUG updating application "target" scale to 1
controller-0: 2026-05-26 01:16:54 DEBUG updating application "target" trust to true
controller-0: 2026-05-26 01:16:54 DEBUG ensuring application "target" exists
controller-0: 2026-05-26 01:16:54 DEBUG application "target" was "unchanged"
controller-0: 2026-05-26 01:16:54 DEBUG application "target" is ready
```

After `01:16:54Z` the provisioner produces no further log entries for this application for the remaining 15+ minutes until the test timeout.

#### Unit agent output during fresh deploy window

There are **zero** `unit-target-0` log entries in `debug-log.txt` between `01:16:54Z` and `01:32:23Z` (the full 15-minute wait window). The unit agent never connects to the Juju controller.

#### Application status log

The `status-log/application-target.txt` for the fresh-deploy application shows a final entry at `01:16:59Z`. No status updates are recorded for the remainder of the test.

#### Kubernetes StatefulSet events

From `statefulset/describe-target.txt` (collected at ~01:33Z):

```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 20m (x5 over 50m) statefulset-controller create Pod target-0 in StatefulSet target successful
Normal SuccessfulDelete 17m (x4 over 27m) statefulset-controller delete Pod target-0 in StatefulSet target successful
```

The most recent `SuccessfulCreate` event is approximately 20 minutes before collection (i.e., during the fresh deploy window). The corresponding `SuccessfulDelete` is approximately 17 minutes before collection (~3 minutes after the create). At log collection time the StatefulSet shows `Replicas: 0 desired | 0 total`.

**No pod describe or container logs for the failing `target-0` pod were available** — the pod was gone before log collection ran.

#### Contrast: `test_downgrade_charm` (rev 12 via `juju refresh`, same execution, PASSES)

In the same execution, `test_downgrade_charm` performs `juju refresh target --revision 12 --channel latest/edge` from rev 19. The unit agent reconnects successfully in the new pod:

```
unit-target-0: 2026-05-26 01:10:33 INFO running containerAgent [4.0.5 ...]
unit-target-0: 2026-05-26 01:10:33 INFO "unit-target-0" successfully connected to "wss://10.152.183.100:17070"
unit-target-0: 2026-05-26 01:10:33 DEBUG container "sloth": pebbleNoticer starting
unit-target-0: 2026-05-26 01:10:45 DEBUG [AGENT-STATUS] executing: running sloth-pebble-ready hook
```

The application reaches `active` at `01:12:46Z` with workload version `0.15.0`. The rev 12 sloth container pebble socket is available immediately on pod start.

---

### Reproduction

This failure occurs in every execution of the test plan against `sloth-k8s` rev 19 on the `juju:4/stable ubuntu:24.04 kubernetes` environment. No failures have been observed on Juju 3.x environments.

Steps the test performs:
1. Deploy `sloth-k8s` rev 19 + `loki-k8s` rev 217 (passes — `test_deploy`)
2. Run `juju refresh target --revision 12` (passes — `test_downgrade_charm`)
3. Remove `target` application
4. Freshly deploy `sloth-k8s` rev 12 from a generated bundle (fails — `test_deploy_target_old_revision`)

---

### Affected test executions (all on `juju:4/stable ubuntu:24.04 kubernetes`, Juju 4.0.5)

| Execution | Test Observer UI | CI run |
|---|---|---|
| 522676 | https://test-observer.canonical.com/#/charms/407793?testExecutionId=522676&testResultId=11223524 | https://github.com/canonical/charm-integration-testing/actions/runs/26425728707 |
| 522698 | https://test-observer.canonical.com/#/charms/407793?testExecutionId=522698&testResultId=11225324 | — |
| 522708 | https://test-observer.canonical.com/#/charms/407793?testExecutionId=522708&testResultId=11224712 | — |
| 522728 | https://test-observer.canonical.com/#/charms/407793?testExecutionId=522728&testResultId=11223830 | — |
| 522742 | https://test-observer.canonical.com/#/charms/407793?testExecutionId=522742&testResultId=11224226 | — |

---

### Diagnostics that would help investigate further

The key missing evidence is the state of the `target-0` pod during the ~3-minute window between its creation and deletion. Capturing the following while the pod is in its failure state (i.e., between fresh deploy and the 15-minute timeout) would help identify the root cause:

1. **Pod describe** — shows container states (e.g. `Init:CrashLoopBackOff`, `ErrImagePull`, `CrashLoopBackOff`), restart counts, kubelet events, and whether the init container (`charm-init`) ever completed:
```
kubectl describe pod target-0 -n
```

2. **Init container logs** — output from the `charm-init` container (which sets up the Juju agent config):
```
kubectl logs target-0 -c charm-init -n
kubectl logs target-0 -c charm-init -n --previous
```

3. **Charm container logs** — output from the `charm` container (pebble + unit agent):
```
kubectl logs target-0 -c charm -n
```

4. **Sloth container logs**:
```
kubectl logs target-0 -c sloth -n
```

---

### Additional notes

- The `sloth-k8s` rev 12 OCI image (`registry.jujucharms.com/charm/76ya4jgjzc7o5sesyuuy5z1eet5pubvsrtxhl/sloth-image@sha256:23878bf05651ddaf0c317f4a8431887fd69d859cc51b822e0fa77ec8f8cb0adf`) is present and functioning correctly when deployed via `juju refresh` — this rules out the image itself being unavailable.
- The failure pattern is exclusive to Juju 4.0.5 (`juju:4/stable`). No failures of this test have been observed on Juju 3.x in Test Observer.
- At the time of log collection the pod `target-0` had already been deleted and was not available for inspection.

Start with test_deploy_target_old_revision in the integration/loki-k8s:logging/loki_push_api/sloth-k8s:logging plan and reproduce it on Juju 4.0.5. Capture kubectl describe pod target-0 and logs from charm-init, charm, and sloth before the pod is deleted, then compare with the passing refresh path and Juju 3.x. Done means identifying the fresh-deploy failure cause and defining a verified fix or regression test.

Written by the indexing model from the issue text.