hashicorp / hashicorp/consul

CNAME resolution of AWS Aurora database cluster address

Open
#3,060 11 comments 0 reactions 0 assignees View on GitHub
theme/api type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

### `consul version` for both Client and Server
Client: `v0.8.2` and `v0.8.3`
Server: `v0.8.2` and `v0.8.3`

### Operating system and Environment details
CentOS7 x86_64

### Description of the Issue (and unexpected/desired result)

Originally posted in the [mailing list](https://groups.google.com/d/msg/consul-tool/x-eceQxHP-U/REiH8IbOBAAJ).

The Aurora database service provides an address for each specific database instance and a _cluster address_ (a CNAME pointing to the current database leader/writer).

Resolution of this cluster address through Consul stops with a CNAME instead of an A record.

This is on an otherwise working Consul cluster using dnsmasq and the AWS VPC DNS recursor.

### Reproduction steps

1. Launch a VPC in AWS
2. Launch an [Aurora RDS cluster](https://aws.amazon.com/rds/aurora/)
3. Launch an EC instance and configure consul and dnsmasq
4. Register 2 services with Consul catalog:

A "db" service in Consul configured with the **instance** address/endpoint:
```
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.43.amzn1 <<>> db.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 487
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;db.service.consul. IN A

;; ANSWER SECTION:
db.service.consul. 5 IN CNAME db.XXXXXXXXXXXX.us-west-2.rds.amazonaws.com.
db.XXXXXXXXXXXX.us-west-2.rds.amazonaws.com. 5 IN CNAME ec2-XXX-XXX-XXX-XXX.us-west-2.compute.amazonaws.com.
ec2-XXX-XXX-XXX-XXX.us-west-2.compute.amazonaws.com. 20 IN A 10.XXX.XXX.XXX
```

A "db" service in Consul configured with the **cluster** address/endpoint:
```
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.43.amzn1 <<>> db.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48735
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;db.service.consul. IN A

;; ANSWER SECTION:
db.service.consul. 5 IN CNAME db-cluster.cluster-XXXXXXXXXXXX.us-west-2.rds.amazonaws.com.
db-cluster.cluster-XXXXXXXXXXXX.us-west-2.rds.amazonaws.com. 5 IN CNAME db.XXXXXXXXXXXX.us-west-2.rds.amazonaws.com.
db.XXXXXXXXXXXX.us-west-2.rds.amazonaws.com. 5 IN CNAME ec2-XXX-XXX-XXX-XXX.us-west-2.compute.amazonaws.com.
```

### Expected result:
Clients can connect via the consul dns using either configuration.

### Actual result:
Clients can connect via the consul dns using the first configuration.

Clients cannot connect via the consul dns using this second configuration due to the missing A record.

### Workaround:

Using the first configuration works for now, but misses out on the benefit of automatic failover of the leader/writer instance.

The Aurora database service also offers a cluster "reader" address/endpoint that provides a CNAME over a pool of read-only instances. While I have not tried it, I suspect it will have DNS resolution issues as the writer address.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.