canonical / canonical/charm-integration-testing
prometheus_scrape validator's http_probe/scrape check fails with CERTIFICATE_VERIFY_FAILED on HTTPS metrics endpoints
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa
### Summary
Test executions across multiple unrelated charms fail the `prometheus_scrape` interface validator's `http_probe`/scrape checks with SSL certificate verification errors, when the target scrape endpoint is served over HTTPS.
### Observed instances
| Test plan | Test Observer link | Result | Error |
|---|---|---|---|
| `vault-k8s:metrics-endpoint/prometheus_scrape/grafana-agent-k8s:metrics-endpoint` | [execution 668396](https://test-observer.canonical.com/#/charms/409648?testExecutionId=668396&testResultId=12668607) | FAILED | `Check 'http_probe' failed: https://10.1.3.26:8200/v1/sys/metrics: ` |
| `postgresql-k8s:metrics-endpoint/prometheus_scrape/prometheus-k8s:metrics-endpoint` | [execution 653290](https://test-observer.canonical.com/#/charms/409527?testExecutionId=653290&testResultId=12328193) | FAILED | `scrape[:XXX]: Scrape failed: ` |
| `vault-k8s:metrics-endpoint/prometheus_scrape/prometheus-k8s:metrics-endpoint` | [execution 653925](https://test-observer.canonical.com/#/charms/409530?testExecutionId=653925&testResultId=12388641) | FAILED | `Check 'http_probe' failed: https://:XXX/v1/sys/metrics: ` |
Note the `postgresql-k8s` instance involves no `vault-k8s` charm — `postgresql-k8s` is validated against its own metrics endpoint independently, and hits the identical `CERTIFICATE_VERIFY_FAILED` signature.
Across Test Observer, at least 4 charms (`vault-k8s`, `postgresql-k8s`, `prometheus-k8s`, `grafana-agent-k8s`) have hit this exact failure signature on `prometheus_scrape` validation in recent runs.
### Steps to reproduce
1. Deploy a charm bundle where a `prometheus_scrape` provider serves its metrics endpoint over HTTPS with a self-signed/internal certificate.
2. Run integration tests with charm-integration-testing's validators at `level=deep`.
3. Observe `http_probe` (L1) or `scrape[...]` (L2) checks fail with `CERTIFICATE_VERIFY_FAILED`.
Contributor guide
Research direction
Start by locating the prometheus_scrape validator entry points used at level=deep, especially the http_probe (L1) and scrape (L2) checks. Reproduce against an HTTPS endpoint with a self-signed or internal certificate, then determine the intended certificate handling and add coverage showing that both checks complete without the reported CERTIFICATE_VERIFY_FAILED error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, python
- Domain
- observability-sre, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100