linkedin / linkedin/Burrow

tls option for AWS MSK

Open
#746 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

I am using AWS MSK with SSL enabled and I need to know how to create the tls cert and key file from keystore. We are using keystore to connect to AWS MSK.

tls burrow config:

```
[tls.name]
certfile="/etc/kafkakeys/orderhistory/client_cert.pem"
keyfile="/etc/kafkakeys/orderhistory/private_key.pem"
cafile="/etc/kafkakeys/orderhistory/truststore.pem"
noverify=false
```
Logs:
```
Reading configuration from /etc/burrow
{"level":"info","ts":1646032362.8147514,"msg":"Started Burrow"}
{"level":"info","ts":1646032362.814817,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1646032362.8156712,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1646032362.8157008,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"mystorage"}
{"level":"info","ts":1646032362.8162286,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1646032362.8162525,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1646032362.8162856,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1646032362.8164155,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1646032363.0167787,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1646032363.016819,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"orderhistory"}
{"level":"panic","ts":1646032363.0298462,"msg":"cannot read TLS certificate or key file: tls: failed to parse private key"}
panic: cannot read TLS certificate or key file: tls: failed to parse private key [recovered]
panic: cannot read TLS certificate or key file: tls: failed to parse private key [recovered]
panic: cannot read TLS certificate or key file: tls: failed to parse private key

goroutine 1 [running]:
main.handleExit()
/usr/src/Burrow/main.go:64 +0x21d
panic(0xc673e0, 0xc0004614f0)
/usr/local/go/src/runtime/panic.go:969 +0x175
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0000dd760, 0x0, 0x0, 0x0)
/go/pkg/mod/go.uber.org/zap@v1.15.0/zapcore/entry.go:230 +0x55f
go.uber.org/zap.(*Logger).Panic(0xc00007a240, 0xc0003b6b40, 0x49, 0x0, 0x0, 0x0)
/go/pkg/mod/go.uber.org/zap@v1.15.0/logger.go:225 +0x85
github.com/linkedin/Burrow/core.configureCoordinators.func1(0xc00007a060)
/usr/src/Burrow/core/burrow.go:97 +0x7d
panic(0xc673e0, 0xc0004614e0)
/usr/local/go/src/runtime/panic.go:969 +0x175
github.com/linkedin/Burrow/core/internal/helpers.GetSaramaConfigFromClientProfile(0xc0001e5bb0, 0xc, 0x23)
/usr/src/Burrow/core/internal/helpers/sarama.go:115 +0xf57
github.com/linkedin/Burrow/core/internal/cluster.(*KafkaCluster).Configure(0xc00028eb00, 0xc00022fd60, 0xc, 0xc0001aa740, 0x14)
/usr/src/Burrow/core/internal/cluster/kafka_cluster.go:63 +0x133
github.com/linkedin/Burrow/core/internal/cluster.(*Coordinator).Configure(0xc00000e340)
/usr/src/Burrow/core/internal/cluster/coordinator.go:83 +0x22b
github.com/linkedin/Burrow/core.configureCoordinators(0xc00007a060, 0xf25d00, 0xc000078100, 0xf25cc0, 0xc000032420, 0xf25a80, 0xc00000e2c0, 0xf25b80, 0xc000032510, 0xf25bc0, ...)
/usr/src/Burrow/core/burrow.go:104 +0xc3
github.com/linkedin/Burrow/core.Start(0xc00007a060, 0xc0000bf800, 0x0)
/usr/src/Burrow/core/burrow.go:152 +0x418
main.main()
/usr/src/Burrow/main.go:115 +0x695
```

Related Issue : [https://github.com/Shopify/sarama/issues/1799]
can someone point me to a doc?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the TLS configuration handling in core/internal/helpers/sarama.go and the related Sarama issue linked in the report. Determine how keystore-derived certificate and key files are expected to be supplied, then document the supported conversion or configuration path and verify it against the reported private-key parsing error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
backend, security
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.