hashicorp / hashicorp/consul

Flagger support for Consul

Open
#7,535 8 comments 8 reactions 0 assignees View on GitHub
needs-discussion theme/connect theme/kubernetes type/docs type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Feature Description

I would like to install one helm chart in several DCs and get Consul Connect service mesh out of the box.

#### Use Case(s)

* multi-DC failover
* canary deployments

#### My experience

I spent a few weekends to make this work on PoC level. And here is my experience.

Problems I faced:
* standard helm chart doesn't support multi-DC scenario
* there are no tools for canary deployments
* not clear procedure to create common trust domain
* multi-DC failover doesn't work without health-check
* exposing health-check in Kubernetes doesn't work
* standard statistics exposed through statsd is limited

Didn't find in docs:
* how to generate SPIFFE SVID (thus used Vault, although it might be overkill for some cases)
* mention that ClusterID and thus trust domain should be same between all DCs

Modified:
* [consul-k8s](https://github.com/hashicorp/consul-k8s/compare/wan-federation-base...oleksiyp:wan-federation-base) project to expose HTTP health checks
* [consul-helm](https://github.com/hashicorp/consul-helm/compare/master...oleksiyp:master) project to make join WAN possible
* distribution of servers and clients by corresponding node labels
* expose hostPorts
* [flagger](https://github.com/weaveworks/flagger/compare/master...oleksiyp:master) to support Consul Connect Canary deployment and multi-DC failover

I added to service chart additional envoy instance to expose statistics to Prometheus directly through envoys /metrics (although now I realize that I potentially can just expose this path)

Created test-bed:
* simple ingress forwarding traffic from outside: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/ingress-chart
* Kubernetes controller called prefixrouter to forward traffic to services based on CRD definitions: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/prefixrouter-chart
* Service instance which randomly picks up next service-to-service call and does a fixed amount of hops: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/service-ktor
* UI client that shows how traffic is distributed: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/ingress-js

As a result:
* I joined Consul Connect DCs in two Kubernetes clusters([kind](https://kind.sigs.k8s.io/docs/user/quick-start))
* I exposed health-check and have an ability to create liveness/readiness probe
* I am forwarding traffic through mesh-gateways by sharing ClusterID
* I am able to perform Flagger deployments and shift traffic from old version to new version
* based on exposed health-checks traffic is redirected to other DC in case of failure

All setup available here: https://github.com/oleksiyp/consul-connect-multi-dc

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.