Regression in Logstash FIPS tests: filebeat panics during TLS 1.3 handshake with OpenSSL 3.6.2
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
## Summary
Filebeat FIPS 8.19.15 crashes with a goroutine panic on every TLS 1.3 connection. This breaks the Filebeat → Logstash → ES pipeline in the observability SRE acceptance tests.
```
panic: failed to add parameter data: OSSL_PARAM_BLD_push_octet_string
openssl error(s):
error:078C0102:common libcrypto routines::passed a null parameter
goroutine 88 [running]:
crypto/tls/internal/tls13.ExpandLabel[...]
crypto/tls/internal/tls13/tls13.go:40
crypto/tls.(*cipherSuiteTLS13).trafficKey(...)
crypto/tls/key_schedule.go:28
crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(...)
crypto/tls/handshake_client_tls13.go:518
```
## Root Cause
OpenSSL 3.6.2 added a NULL check to `OSSL_PARAM_BLD_push_octet_string` that rejects nil buffers. Go's TLS 1.3 `ExpandLabel` passes a nil context through `golang-fips/openssl`, which now panics instead of being treated as empty.
## Existing Upstream Issues
- **OpenSSL regression**: https://github.com/openssl/openssl/issues/30728 — fix merged in https://github.com/openssl/openssl/pull/30730
- **golang-fips/openssl workaround**: https://github.com/golang-fips/openssl/pull/417
## Reproduction
The Logstash observability SRE acceptance tests expose this via the [filebeat-to-ls-to-es.conf](https://github.com/elastic/logstash/blob/8.19/x-pack/distributions/internal/observabilitySRE/qa/acceptance/docker/logstash/pipeline/filebeat-to-ls-to-es.conf) pipeline, which configures a Beats input with TLS:
```bash
export OBSERVABILITY_SRE_IMAGE_VERSION="8.19.15-SNAPSHOT"
export ELASTICSEARCH_IMAGE_VERSION="8.19.15-SNAPSHOT"
export FILEBEAT_IMAGE_VERSION="8.19.15-SNAPSHOT"
./gradlew observabilitySREacceptanceTests --stacktrace
```
3 of 4 tests pass. The "Filebeat through LS to ES in a FIPS compliant configuration" test fails because Filebeat panics before sending any data.
## Environment
| Component | Version |
|-----------|---------|
| Filebeat | 8.19.15-SNAPSHOT FIPS (`elasticsearch-cloud-ess-fips`) |
| Go | 1.25.8 |
| OpenSSL | 3.6.2 |
| Logstash | 8.19.15-SNAPSHOT observability-SRE image |
Contributor guide
Research direction
Start with the observability SRE acceptance command and the pipeline in x-pack/distributions/internal/observabilitySRE/qa/acceptance/docker/logstash/pipeline/filebeat-to-ls-to-es.conf. Trace the Filebeat FIPS TLS 1.3 handshake and compare the reported nil-buffer behavior with the linked golang-fips/openssl workaround. Done means the FIPS acceptance test completes without a panic and the Filebeat-to-Logstash-to-Elasticsearch pipeline sends data successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking, observability-sre, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100