elastic / elastic/beats

[Heartbeat] Silent remote error: tls: handshake failure

Open
#48,104 1 comment 1 reaction 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

After upgrading Filebeat to 8.19.2, HTTP monitors configured with `ssl.verification_mode: none` incorrectly report as down, and the API `/internal/uptime/monitor/status?monitorId=&dateStart=now-15m&dateEnd=now` returns the following error:

```json
"error": {
"message": "Get \"https://example.com:80/healthcheck\": remote error: tls: handshake failure",
"type": "io"
},
"url": {
"path": "/healthcheck",
"full": "https://example.com:80/healthcheck",
"scheme": "https",
"domain": "example.com",
"port": 80
}
```

There is nothing relevant in the logs, even at debug level. We could track it down to these changes:

- https://github.com/golang/go/issues/66512#issuecomment-2019511690
>In Go 1.22 RSA key exchange based cipher suites were removed from the default list, but can be re-added with the GODEBUG setting tlsrsakex=1.
- https://github.com/elastic/beats/pull/40082#issuecomment-2227716350
>By default, cipher suites without ECDHE support are no longer offered by either clients or servers during pre-TLS 1.3 handshakes. This change can be reverted with the tlsrsakex=1 GODEBUG setting.

While setting the `GODEBUG=tlsrsakex=1` environment variable resolves the issue, the error is really not obvious as Heartbeat doesn't log anything.

Possible ways to address this bug include:
- Logging the TLS handshake issue at the error level and providing guidance on how to resolve it.
- Documenting this breaking change in the official documentation.
- Alternatively, setting `GODEBUG=tlsrsakex=1` by defautl

Contributor guide

Open the contributing guide

Research direction

Start at Heartbeat's HTTP monitor handling for the reported TLS handshake failure and reproduce the example with ssl.verification_mode: none against an endpoint using the affected cipher negotiation. Compare the behavior with GODEBUG=tlsrsakex=1; done should be an agreed fix that makes the failure visible and actionable, documents the compatibility change, or explicitly chooses another listed remedy.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.