Flagger support for Consul
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Feature Description
I would like to install one helm chart in several DCs and get Consul Connect service mesh out of the box.
#### Use Case(s)
* multi-DC failover
* canary deployments
#### My experience
I spent a few weekends to make this work on PoC level. And here is my experience.
Problems I faced:
* standard helm chart doesn't support multi-DC scenario
* there are no tools for canary deployments
* not clear procedure to create common trust domain
* multi-DC failover doesn't work without health-check
* exposing health-check in Kubernetes doesn't work
* standard statistics exposed through statsd is limited
Didn't find in docs:
* how to generate SPIFFE SVID (thus used Vault, although it might be overkill for some cases)
* mention that ClusterID and thus trust domain should be same between all DCs
Modified:
* [consul-k8s](https://github.com/hashicorp/consul-k8s/compare/wan-federation-base...oleksiyp:wan-federation-base) project to expose HTTP health checks
* [consul-helm](https://github.com/hashicorp/consul-helm/compare/master...oleksiyp:master) project to make join WAN possible
* distribution of servers and clients by corresponding node labels
* expose hostPorts
* [flagger](https://github.com/weaveworks/flagger/compare/master...oleksiyp:master) to support Consul Connect Canary deployment and multi-DC failover
I added to service chart additional envoy instance to expose statistics to Prometheus directly through envoys /metrics (although now I realize that I potentially can just expose this path)
Created test-bed:
* simple ingress forwarding traffic from outside: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/ingress-chart
* Kubernetes controller called prefixrouter to forward traffic to services based on CRD definitions: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/prefixrouter-chart
* Service instance which randomly picks up next service-to-service call and does a fixed amount of hops: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/service-ktor
* UI client that shows how traffic is distributed: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/ingress-js
As a result:
* I joined Consul Connect DCs in two Kubernetes clusters([kind](https://kind.sigs.k8s.io/docs/user/quick-start))
* I exposed health-check and have an ability to create liveness/readiness probe
* I am forwarding traffic through mesh-gateways by sharing ClusterID
* I am able to perform Flagger deployments and shift traffic from old version to new version
* based on exposed health-checks traffic is redirected to other DC in case of failure
All setup available here: https://github.com/oleksiyp/consul-connect-multi-dc
Contributor guide
Assessment
This issue has not been assessed yet.