aws / aws/containers-roadmap

[service] [request]: Deploy Multiple Instances of Flink Kubernetes Operator for EMR on EKS in One EKS Cluster

Open
#2,316 0 comments 0 reactions 0 assignees View on GitHub
EKS EMR Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
What do you want us to build?
For multi-tenant cluster, dedicated operators should be deployed for each namespace, and multiple operators may be deployed in one EKS cluster; however, the Flink Kubernetes operator for EMR on EKS, i.e., [https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/jobruns-flink-kubernetes-operator-getting-started.html ](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/jobruns-flink-kubernetes-operator-getting-started.html), seems cluster scoped, which allows only a single instance to be deployed in one EKS cluster regardless of the namespaces they are deployed to.

We've tried the following operator deployment in one EKS cluster:
After deploying one instance of the operator for EMR on EKS in the namespace x, deploying the second instance in the namespace y got the error that the cluster role already existed in the cluster, which was already created by the first instance.

For multi-tenant cluster, we'd like to request that the Flink Kubernetes operator for EMR on EKS should support multiple instances in one EKS cluster like Google one [https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/master/docs/user_guide.md#:~:text=How%2Dto-,Deploy%20multiple%20instances%20of%20the%20operator%20in%20a%20cluster,-The%20Flink%20operator ](https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/master/docs/user_guide.md#:~:text=How%2Dto-,Deploy%20multiple%20instances%20of%20the%20operator%20in%20a%20cluster,-The%20Flink%20operator).

**Which service(s) is this request for?**
This could be Fargate, ECS, EKS, ECR
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

**Are you currently working around this issue?**
How are you currently solving this problem?

**Additional context**
Anything else we should know?

**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the reported ClusterRole conflict and the linked Google Cloud Flink operator guide on deploying multiple instances. Define the deployment and namespace-scoping requirements for EMR on EKS, with done meaning separate operator instances can coexist in one EKS cluster without resource conflicts.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.