canonical / canonical/charm-integration-testing

prometheus_scrape validator's http_probe/scrape check fails with CERTIFICATE_VERIFY_FAILED on HTTPS metrics endpoints

Open
#836 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 20h
Merged PRs (30d)
98

Description

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

### Summary
Test executions across multiple unrelated charms fail the `prometheus_scrape` interface validator's `http_probe`/scrape checks with SSL certificate verification errors, when the target scrape endpoint is served over HTTPS.

### Observed instances
| Test plan | Test Observer link | Result | Error |
|---|---|---|---|
| `vault-k8s:metrics-endpoint/prometheus_scrape/grafana-agent-k8s:metrics-endpoint` | [execution 668396](https://test-observer.canonical.com/#/charms/409648?testExecutionId=668396&testResultId=12668607) | FAILED | `Check 'http_probe' failed: https://10.1.3.26:8200/v1/sys/metrics: ` |
| `postgresql-k8s:metrics-endpoint/prometheus_scrape/prometheus-k8s:metrics-endpoint` | [execution 653290](https://test-observer.canonical.com/#/charms/409527?testExecutionId=653290&testResultId=12328193) | FAILED | `scrape[:XXX]: Scrape failed: ` |
| `vault-k8s:metrics-endpoint/prometheus_scrape/prometheus-k8s:metrics-endpoint` | [execution 653925](https://test-observer.canonical.com/#/charms/409530?testExecutionId=653925&testResultId=12388641) | FAILED | `Check 'http_probe' failed: https://:XXX/v1/sys/metrics: ` |

Note the `postgresql-k8s` instance involves no `vault-k8s` charm — `postgresql-k8s` is validated against its own metrics endpoint independently, and hits the identical `CERTIFICATE_VERIFY_FAILED` signature.

Across Test Observer, at least 4 charms (`vault-k8s`, `postgresql-k8s`, `prometheus-k8s`, `grafana-agent-k8s`) have hit this exact failure signature on `prometheus_scrape` validation in recent runs.

### Steps to reproduce
1. Deploy a charm bundle where a `prometheus_scrape` provider serves its metrics endpoint over HTTPS with a self-signed/internal certificate.
2. Run integration tests with charm-integration-testing's validators at `level=deep`.
3. Observe `http_probe` (L1) or `scrape[...]` (L2) checks fail with `CERTIFICATE_VERIFY_FAILED`.

Contributor guide

Open the contributing guide

Research direction

Start by locating the prometheus_scrape validator entry points used at level=deep, especially the http_probe (L1) and scrape (L2) checks. Reproduce against an HTTPS endpoint with a self-signed or internal certificate, then determine the intended certificate handling and add coverage showing that both checks complete without the reported CERTIFICATE_VERIFY_FAILED error.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
observability-sre, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.