hashicorp / hashicorp/consul-template

Simultaneous SSL update on all instances.

Open
#1,704 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4.8k
Forks
801
Avg merge
4h 5m
Merged PRs (30d)
6

Description

Hi!

We use consul-template + Vault PKI to provide SSL certificates for the MySQL Galera cluster. We did some tests with short TTL (15m) for SSL and faced the issue when the Galera cluster crashed because of simultaneous SSL re-generation for all nodes (we send `ALTER INSTANCE RELOAD TLS;` via reload-script each time a new certificate has been done).

Also, we faced the same issue with the Apache Kafka cluster (with SSL) but TTL was 7 days. Honestly, it was only once for 1 month, but it has happened.

We applied a fix to shift TTL for 1 day for every next node, it helps to reduce the chance, but it's not a fix.

My question is simple: Any way you have some distributive lock (via Consul) to prevent all instances from updating certificates at the same time?

### mysqld config x 3 instatces

```
$ cat my.cnf
[client]
port = 33306
socket = /tmp/mysql.sock
default-character-set = utf8

[mysqld]
pxc_encrypt_cluster_traffic=ON
user = mysql
ssl-ca = /opt/mysql/tls/server/server-ca.pem
ssl-cert = /opt/mysql/tls/server/server-cert.pem
ssl-key = /opt/mysql/tls/server/server-key.pem
...

```

### consul-template configs x 3 instances

```
$ cat conf/consul-template.hcl
vault {
address = "https://127.0.0.1:8200"

unwrap_token = false
renew_token = true

lease_renewal_threshold = 0.5

ssl {
enabled = true
verify = true
ca_path = "/opt/consul-template/tls/server-CA.cert"
cert = "/opt/consul-template/tls/consul-template.cert"
key = "/opt/consul-template/tls/consul-template.key"
server_name = "127.0.0.1"
}
}

# MYSQL
template {
source = "/opt/consul-template/templates/mysql/server-ca.pem.tpl"
destination = "/opt/mysql/tls/server/server-ca.pem"
perms = 0640
command = "/opt/consul-template/templates/mysql/reload.sh"
error_on_missing_key = true
left_delimiter = "[["
right_delimiter = "]]"
}
template {
source = "/opt/consul-template/templates/mysql/server-cert.pem.tpl"
destination = "/opt/mysql/tls/server/server-cert.pem"
perms = 0640
command = "/opt/consul-template/templates/mysql/reload.sh"
error_on_missing_key = true
left_delimiter = "[["
right_delimiter = "]]"
}
template {
source = "/opt/consul-template/templates/mysql/server-key.pem.tpl"
destination = "/opt/mysql/tls/server/server-key.pem"
perms = 0640
command = "/opt/consul-template/templates/mysql/reload.sh"
error_on_missing_key = true
left_delimiter = "[["
right_delimiter = "]]"
}

```
+
```
$ cat /opt/consul-template/templates/mysql/reload.sh
#!/bin/bash

set -eo pipefail

STATUS=0

if [ -f '/opt/mysql/current/bin/mysql' -a -S '/tmp/mysql.sock' ];
then
echo "ALTER INSTANCE RELOAD TLS;" | /opt/mysql/current/bin/mysql -u root -p'super-secure-password'
STATUS=$?
fi

exit $STATUS
```

Contributor guide

Open the contributing guide

Research direction

Start with the consul-template configuration and /opt/consul-template/templates/mysql/reload.sh shown in the issue, then inspect the project for existing Consul coordination or locking entry points. Done would require a defined mechanism to prevent simultaneous certificate reloads across instances, along with project tests or documentation demonstrating the behavior; no repository files or tests are identified here.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, kafka, mysql
Domain
databases, devops, distributed-systems, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.