Add cleanup of stale istio ValidatingWebhookConfigurations between test runs

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

---

### Problem Summary

Istio-based charm tests (istio-pilot, istio-gateway, and charms that depend on them) consistently fail on shared Kubernetes test clusters due to orphaned `ValidatingWebhookConfiguration` resources left behind from previous test runs. This is a workaround for upstream issue [canonical/istio-operators#551](https://github.com/canonical/istio-operators/issues/551) (to be linked after filing).

**Test Observer:** https://test-observer.canonical.com/#/charms/316242?testExecutionId=315890&testResultId=8822949

**Affected test plans:** All integration tests involving:
- istio-pilot
- istio-gateway
- dex-auth (depends on istio-pilot)
- oidc-gatekeeper (depends on istio-pilot)
- tensorboard-controller (depends on istio-pilot)

**Current failure rate:** 100% for dex-auth revision 783 (10 consecutive failures)

---

### Root Cause

The istio-pilot charm creates cluster-scoped `ValidatingWebhookConfiguration` resources but does not clean them up during model destruction (upstream charm bug). When parallel tests run on the same Kubernetes cluster:

1. Test execution A creates webhook → test completes → model destroyed
2. Webhook configuration remains in cluster, points to deleted namespace
3. Test execution B starts → Kubernetes invokes stale webhook
4. TLS validation fails with x509 errors → test fails

**Error from test logs:**
```
lightkube.core.exceptions.ApiError: failed calling webhook "validation.istio.io":
Post "https://istiod.model-21229244971-260121231735.svc:443/validate?timeout=10s":
tls: failed to verify certificate: x509: certificate signed by unknown authority
```

---

### Impact on CI/CD

**All integration tests using istio-pilot fail on shared clusters:**
- dex-auth: 10/10 failures (revision 783)
- Tests timeout waiting for istio-pilot to reach active state
- Parallel test executions interfere with each other

**Evidence:** Test execution 315890 shows the new istiod pod managing 6 different webhook configurations (including 5 from previous test runs).

---

### Proposed Solution

Add a cleanup step to the test framework to delete stale istio webhook configurations **before** each test run or **after** model destruction.

**Option 1: Pre-test cleanup (Recommended)**

Add to test initialization phase:
```python
def cleanup_stale_istio_webhooks(k8s_client):
"""Remove orphaned istio webhook configurations from previous test runs."""
try:
# Delete stale validating webhook configurations
k8s_client.delete_collection(
group="admissionregistration.k8s.io",
version="v1",
plural="validatingwebhookconfigurations",
label_selector="app=istiod",
# Only delete webhooks pointing to non-existent namespaces
field_selector="metadata.name~=istio-validator-*"
)
logger.info("Cleaned up stale istio webhook configurations")
except Exception as e:
logger.warning(f"Webhook cleanup failed (non-critical): {e}")
```

**Option 2: Post-test cleanup**

Add to model teardown phase:
```bash
# After destroying the Juju model
MODEL_UUID=$(juju show-model $MODEL_NAME --format=json | jq -r '.[].model-uuid')
kubectl delete validatingwebhookconfigurations istio-validator-model-${MODEL_UUID} --ignore-not-found=true
```

**Option 3: Periodic cleanup job**

Add a Kubernetes CronJob to the test cluster:
```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: cleanup-stale-istio-webhooks
spec:
schedule: "*/30 * * * *" # Every 30 minutes
jobTemplate:
spec:
template:
spec:
serviceAccountName: webhook-cleaner
containers:
- name: cleanup
image: bitnami/kubectl:latest
command:
- /bin/bash
- -c
- |
# Delete webhooks pointing to non-existent namespaces
kubectl get validatingwebhookconfigurations -l app=istiod -o json | \
jq -r '.items[] | select(.webhooks[0].clientConfig.service.namespace |
test("^model-")) | .metadata.name' | \
while read webhook; do
namespace=$(kubectl get validatingwebhookconfigurations $webhook -o jsonpath='{.webhooks[0].clientConfig.service.namespace}')
if ! kubectl get namespace $namespace &>/dev/null; then
echo "Deleting stale webhook $webhook (namespace $namespace no longer exists)"
kubectl delete validatingwebhookconfigurations $webhook
fi
done
```

---

### Recommended Approach

**Implement Option 1 (Pre-test cleanup)** as it:
- Runs before tests, preventing failures
- Is non-invasive (only deletes orphaned resources)
- Works immediately without waiting for upstream charm fix
- Can be removed once the charm is fixed

Add this cleanup to the existing Kubernetes setup/initialization code in the test framework.

---

### Verification

After implementing the fix:
1. Run istio-pilot-based integration tests on shared cluster
2. Verify no stale webhook configurations exist: `kubectl get validatingwebhookconfigurations | grep istio-validator`
3. Confirm tests pass consistently
4. Monitor test execution logs for x509 certificate validation errors (should be gone)

---

Start in the existing Kubernetes setup or initialization code in the test framework, and inspect the model teardown path as an alternative entry point. Run the affected istio-pilot-based integration tests and check validating webhook configurations before and after runs. Done means stale Istio webhooks are removed and the x509 failures no longer occur.

Written by the indexing model from the issue text.