hashicorp / hashicorp/consul

Extensible discovery for Cloud Auto-Join

Open
#9,100 4 comments 18 reactions 0 assignees View on GitHub
theme/internal-cleanup theme/internals type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Feature Description

Currently the [cloud auto-join](https://www.consul.io/docs/install/cloud-auto-join) feature allows a user to query a cloud provider API using [go-discover](https://github.com/hashicorp/go-discover) to discover nodes that are part of a Consul cluster.

This feature works well for some use cases, but has a couple limitations:
* discovery only works with the cloud providers known to `go-discovery`. Supporting additional providers requires that `go-discovery` be updated.
* the API queries performed by `go-discovery` are limited to a single tag, there's no way to do a more sophisticated query with multiple tags or any other properties that may be unique to a particular environment.

In addition to the `go-discovery` provider, we could allow the discovery of addresses by any external command. Using a value of `exec=/usr/local/bin/query-ec2` would run the script at that path. The script would exit zero and return one server per line on success, or exit non-zero and print an error message of up to 1024 bytes to stderr on error.

The script could use the cloud provider CLI, or any other command, to perform the query. Since these operations are already performing the API requests, the overhead of starting a new process shouldn't be too significant.

#### Use Case(s)

This approach could potentially solve a number of use cases:

* https://github.com/hashicorp/go-discover/pull/159 - it would allow cloud auto-join to work with new or less widely used clouds without waiting on the code to be merged into go-discover and then Consul.
* https://github.com/hashicorp/consul/issues/6946 - it would allow for more complex queries that don't fit into the single-tag model of `go-discover`.
* It would allow auto-join to work with any custom environment, even those that don't have public APIs or public API clients.

#### Background

`go-discover` also accounts for about [30 of the 107Mb](https://github.com/hashicorp/consul/issues/9074) used by the Consul binary. It also the library that brings in the most number of dependencies, which can make upgrading to new major versions of shared dependencies challenging, and also causes a lot of [false positives on security scans](https://github.com/hashicorp/go-discover/issues/160). None of these issues by themselves are all that significant, but together they do add up. An approach that removes the need to include every provider in the binary would help mitigate these problems.

Other related issues #9754

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.