grafana / grafana/pyroscope

Distributor connects to ingester via plan http despite configured TLS

Open
#4,585 1 comment 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Describe the bug

You can see the code here:

https://github.com/grafana/pyroscope/blob/39e0e7c9f5ee9c61b352ce0cafaf033f79be2205/pkg/clientpool/ingester_client_pool.go#L65

It is using plain HTTP despite me having TLS configured throughout.

In the logs:

```
Oct 31 03:59:30 143dm6 pyroscope[4186086]: http: TLS handshake error from [2400:...2844]:48136: tls: first record does not look like a TLS handshake
```

```
Oct 31 03:59:30 143dm6 pyroscope[4186086]: ts=2025-10-31T03:59:30.77202765Z caller=distributor.go:396 component=distributor tenant=anonymous user=anonymous level=warn msg="profile rejected" service_name=grafana.alloy.ebpf profile_type=process_cpu matched_usage_groups=[] detected_language=go profile_time=2025-10-31T03:59:27.807Z ingestion_delay=2.963s decompressed_size=21244 sample_count=149 err="unavailable: write tcp [2400:...2844]:48136->[2400:...2844]:4040: write: connection reset by peer"
```

```
Oct 31 03:59:30 143dm6 pyroscope[4186086]: ts=2025-10-31T03:59:30.207856516Z caller=distributor.go:396 component=distributor tenant=anonymous user=anonymous level=warn msg="profile rejected" service_name=grafana.alloy.ebpf profile_type=process_cpu matched_usage_groups=[] detected_language=unknown profile_time=2025-10-31T03:59:27.812Z ingestion_delay=2.394s decompressed_size=420 sample_count=1 err="unavailable: unexpected EOF"
```

I can see plaintext queries in tcpdump:

Image

#### To Reproduce
Steps to reproduce the behavior:

1. Configure TLS for all components in a single node deployment.
2. Send some data to the node via HTTPS.
3. Observe complaints about ingestion.

#### Expected behavior

I expect TLS to work.

#### Environment

- Infrastructure: bare metal, single node
- Deployment tool: manual

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.