canonical / canonical/charm-integration-testing

tempo-worker-k8s blocked with "Pebble check [ready] is 'DOWN'." causes test_deploy failures across multiple charms (near-100% since 2026-07-13)

Open
#837 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 20h
Merged PRs (30d)
98

Description

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

### Summary
`tempo-worker-k8s` goes `blocked: Pebble check [ready] is 'DOWN'.` during `test_deploy`, causing the run to time out for multiple, unrelated charm bundles that integrate `tempo-coordinator-k8s`/`tempo-worker-k8s` for tracing.

### Components involved (as provisioned by the harness for these runs)
- `tempo-worker-k8s` rev 117 (2.10/stable)
- `tempo-coordinator-k8s` rev 162 (2.10/stable)
- `s3-integrator` rev 562 (1/stable)
- `minio` (s3-integrator-minio fixture) rev 583 (ckf-1.10/stable)

### Observed failure rate over time
Querying all `test_deploy` results for `tempo-worker-k8s` rev 117 (same charm/minio/s3-integrator revisions throughout):
- 2026-07-07 to 2026-07-13T11:06 — 19 PASSED / 31 FAILED (mixed)
- 2026-07-13T13:47 onward through 2026-08-03 (most recent data at time of triage) — 0 PASSED / 48 FAILED

### Affected artefacts (31 FAILED `test_deploy` results, 2026-07-08 to 2026-08-03)
mysql-k8s, mysql-router-k8s, vault-k8s, grafana-agent-k8s, prometheus-k8s, identity-platform-login-ui-operator, postgresql-k8s — all sharing execution metadata:
`charm_qa:failure:charm:tempo-worker-k8s:status = ["application:blocked:Pebble check [ready] is 'DOWN'.", "unit:blocked:Pebble check [ready] is 'DOWN'."]`

### Exact error messages (verbatim, from `tempo-worker-k8s-0` pod container logs)

Test execution 672250 (mysql-router-k8s/920, 2026-08-03) — https://test-observer.canonical.com/#/charms/409697?testExecutionId=672250&testResultId=12686522:
```
level=error ts=2026-08-03T22:20:41.064750927Z caller=main.go:110 msg="error running Tempo" err="failed to init module services: error initialising module: optional-store: failed to create store: unexpected error from ListObjects on minio-bucket-for-testing: Access Denied."
```

Test execution 653217 (postgresql-k8s, 2026-07-08) — https://test-observer.canonical.com/#/charms/409527?testExecutionId=653217&testResultId=12274523 — repeated 100 times over the run with no successful start:
```
level=error ts=2026-07-08T09:03:45.395717608Z caller=main.go:110 msg="error running Tempo" err="failed to init module services: error initialising module: optional-store: failed to create store: unexpected error from ListObjects on minio-bucket-for-testing: Get \"http://10.1.2.161:9000/minio-bucket-for-testing/?location=\": dial tcp 10.1.2.161:9000: connect: connection refused"
```

K8s pod events, test execution 672250 — https://test-observer.canonical.com/#/charms/409697?testExecutionId=672250&testResultId=12686522:
```
Warning Unhealthy 25m (x16 over 26m) kubelet spec.containers{charm}: Readiness probe failed: HTTP probe failed with statuscode: 502
Warning Unhealthy 98s (x307 over 26m) kubelet spec.containers{tempo}: Readiness probe failed: HTTP probe failed with statuscode: 502
```

Juju status, test execution 672250 — https://test-observer.canonical.com/#/charms/409697?testExecutionId=672250&testResultId=12686522:
```
tempo-worker-k8s 2.10.3 blocked 1 tempo-worker-k8s 2.10/stable 117 10.152.183.203 no Pebble check [ready] is 'DOWN'.
tempo-worker-k8s/0* blocked idle 10.1.3.207 Pebble check [ready] is 'DOWN'.
```

Test comment (all 31 results, example from test execution 672250 — https://test-observer.canonical.com/#/charms/409697?testExecutionId=672250&testResultId=12686522):
```
juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active], units: [active], unit agents: [idle] (applications: ['tempo-worker-k8s'], units: ['tempo-worker-k8s/0'])
```

### Steps to reproduce
1. Run the `charm_integration_testing.test_suite.deploy.test_deploy` suite for any charm bundle whose test plan integrates `tempo-coordinator-k8s`/`tempo-worker-k8s` via the `tracing` relation (e.g. test plan `integration/tempo-coordinator-k8s:tracing/tracing/mysql-router-k8s:tracing`, test execution 672250 — https://test-observer.canonical.com/#/charms/409697?testExecutionId=672250&testResultId=12686522), using the standard harness-provisioned `s3-integrator` + MinIO fixture.
2. Wait for `juju wait-for` to time out (~15 minutes).
3. Observe `tempo-worker-k8s` unit/application status `blocked: Pebble check [ready] is 'DOWN'.`

### Additional example (for cross-reference)
Test execution 653217 (postgresql-k8s, 2026-07-08) — https://test-observer.canonical.com/#/charms/409527?testExecutionId=653217&testResultId=12274523

Contributor guide

Open the contributing guide

Research direction

Start with the charm_integration_testing.test_suite.deploy.test_deploy suite and reproduce the tracing deployment using the harness-provisioned s3-integrator and MinIO fixture. Compare the reported Tempo and readiness-probe errors across the linked test executions, then trace the failure to the relevant test or deployment handling. Done means the affected test_deploy runs complete with tempo-worker-k8s active and ready.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.