[EKS] [request]: Per microservice DNS failover across regions
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
What do you want us to build?
Per microservice aware DNS failover capability with Amazon EKS and Route 53.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.
Many organizations are running microservices on Amazon Elastic Kubernetes Service (EKS). In some cases, there is a platform team that provides Kubernetes as a service through a multi-tenant shared EKS cluster running in its own Virtual Private Cloud (VPC) maintained by the platform team. This shared EKS cluster hosts multiple applications/microservices, each owned by a separate application team within the organization.
While the applications may consist of Pods, they can also include other components such as RDS, Aurora, DynamoDB, S3, Lambda, or EC2 instances. These additional components run in VPCs owned and maintained by the respective application teams. As a result, although the EKS platform is shared across applications, each application has its own reliability, performance, disaster recovery and operational excellence characteristics.
Platform teams often leverage the AWS Load Balancer Controller to instantiate an AWS Application Load Balancer (ALB) as a shared ingress on the EKS cluster. Although the Application Load Balancer is shared across applications, each application should have its own failover capabilities. However, there is a shared fate on the ALB, creating an interesting challenge when organizations implement multi-region architectures for their applications.
To route client requests in a multi-region architecture, organizations use Route 53 routing policies and an alias record per microservice/target group, along with the "evaluate target health" feature. The current "evaluate target health" feature in Route 53 is not target group aware. This means that if one of the target groups (e.g., for microservice#1) no longer has any healthy targets, Route 53 will declare the entire ALB as unhealthy, even though the other target groups (e.g., microservice#2, microservice#3) may still have healthy targets.
This behavior forces customers to either:
1. Fail all microservices in a given region, even though only one has actually failed, or
2. Use individual ALBs per microservice, which can introduce operational complexities and might not be cost-efficient.
[1] [Disaster Recovery (DR) Architecture on AWS: Part IV - Multi-Site Active-Active](https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iv-multi-site-active-active/)
[2] [Choosing Between Alias and Non-Alias Records](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html)
[3] [Target Group Health Checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health.html#r53-dns-failover)
**Are you currently working around this issue?**
How are you currently solving this problem?
Currently you have to implement individual Route 53 healthchecks for each applications URI on ALB and disable evaluate target health feature on DNS records in Route 53. This still increases the rule count on ALB though since per application you need one ALB forwarding rule for the application itself and then another ALB forwarding rule for the application health endpoint.
**Additional context**
Anything else we should know?
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
Contributor guide
Research direction
No repository files, tests, or implementation entry points are identified in the issue. Start by determining whether EKS, the AWS Load Balancer Controller, or Route 53 owns the required behavior; done would mean per-microservice, cross-region DNS failover without requiring a separate ALB for each microservice.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100