canonical / canonical/charm-integration-testing

[Registry] Known cross-charm test-failure signatures (append new findings here)

Open
#924 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 20h
Merged PRs (30d)
96

Description

## For AI agents reading this issue

> **This issue is a registry of known cross-charm test-failure signatures**, not a report about
> one specific charm. If you are investigating a failing test execution for *any* charm, check
> this registry **before** doing your own root-cause investigation — your failure may already be
> a known, documented pattern caused by a *different* charm in the same bundle/test plan.

### How to use this registry

1. Identify the charms present in your failing test execution (`charm_qa:charm` in
`execution_metadata`, or the bundle/test-plan definition).
2. Look up each of those charms against the `signature.trigger_charm` key of every entry under
`findings:` below.
3. For a matching entry, apply its `detection` query against your execution's
`execution_metadata`. If it matches, your target charm's failure is very likely **collateral
damage** from `trigger_charm`, not a bug in the charm you were asked to investigate.
4. If it matches: stop investigating your target charm for that execution. Cite this issue and the
entry's `fix` reference instead, and move on to executions that don't match.
5. If the entry's `fix` has already merged and you still observe the failure, that's a regression —
flag it separately rather than re-deriving the root cause from scratch.
6. **Found a new common/cross-charm failure pattern that isn't listed here?** Append a new entry
to the registry (as a comment on this issue, or a PR updating this issue body) using the same
`signature` / `detection` / `root_cause` / `fix` / `impact` shape as existing entries, so future
agents can look it up the same way.

---

## Registry

```yaml
registry_version: 1
findings:

- id: istio-beacon-k8s-never-active
signature:
trigger_charm: istio-beacon-k8s
trigger_charm_revision: 74
trigger_charm_channel: 1/stable
symptom: "charm crashes on config-changed hook, never reaches active"
detection:
description: >
Query Test Observer's search_test_executions (FAILED, default filters:
artefact_is_archived=false, execution_is_latest=true, rerun_is_requested=false)
for the charm under investigation. For each result, parse execution_metadata
and check all of the following:
keys:
charm_qa:charm: "list includes 'istio-beacon-k8s'"
charm_qa:charm:istio-beacon-k8s:revision: "== '74'"
charm_qa:failure:charm:istio-beacon-k8s:status: "contains 'config-changed'"
match_meaning: "the target charm's failure is very likely attributable to istio-beacon-k8s, not the target charm"
root_cause:
previously_filed_as: canonical/charm-integration-testing#455
previous_diagnosis: "RBAC 403 due to missing `trust: true`"
previous_diagnosis_status: INCORRECT/STALE
actual_root_cause:
- "404 on CRD gateways.gateway.networking.k8s.io (Gateway API CRD not installed in the test model)"
- "no real istio-k8s control plane deployed anywhere in the model/bundle (no Juju relation forces this)"
actual_root_cause_secondary_symptom: >
even with the CRD manually installed, the charm stalls at
maintenance: 'Validating waypoint readiness' because no istio-k8s control
plane exists to talk to
fix:
pr: canonical/charm-integration-testing#922
mechanism: >
declarative cluster_addons/addon_scope override fields + a new
ClusterAddonExtension that deploys istio-k8s (channel 1/stable) as a
cluster-scoped singleton addon before dependent-charm readiness is asserted
files:
- bundle_builder_x/bundle_builder_x/overrides.py
- charm_integration_testing/extensions/cluster_addon/extension.py
- static/charm-overrides/istio-beacon-k8s.yaml
- static/charm-overrides/istio-k8s.yaml
verification_method: >
live sandbox deploy: istio-k8s alone -> active + installs full Gateway API
CRD set as a side effect; istio-beacon-k8s deployed into same model
afterward -> active (previously stuck in maintenance/crash-loop)
impact:
investigated_via_charm: grafana-agent-k8s
note: "impact stats below are a point-in-time snapshot from the investigation that surfaced this entry -- re-run the detection query for current numbers, and expect similar collateral impact on any charm sharing a test plan with istio-beacon-k8s"
snapshot_date: "2026-08-31"
total_failed_executions_for_investigated_charm: 206
executions_matching_signature: 55
executions_matching_signature_pct: "~27%"
distinct_cooccurring_test_plans_or_charms: 20
example_cooccurring_charms:
- minio
- vault-k8s
- kafka-k8s
- mongodb-k8s
- traefik-k8s
- zookeeper-k8s
- parca-k8s
- jupyter-controller
- indico
- hockeypuck-k8s
- opentelemetry-collector-k8s
secondary_cooccurring_issues_not_necessarily_related:
- "#275"
- "#235"
- "#837"
- "istio-beacon-k8s-operator#166"
- "traefik-k8s-operator#366"
- "juju/juju#22724"

# Append additional entries here as new common/cross-charm failure patterns
# are discovered, using the same signature/detection/root_cause/fix/impact shape.
```

## Related

- Superseded diagnosis for the first entry above: #455
- Fix PR for the first entry above: #922

Contributor guide

Open the contributing guide

Research direction

Read the registry instructions and the existing YAML entry in issue #924, then review the linked fix context in canonical/charm-integration-testing#922. Confirm a new cross-charm failure pattern with Test Observer evidence before adding an entry using the documented signature, detection, root_cause, fix, and impact shape; done means future investigations can match and cite it consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, testing-qa
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.