[EKS]: Make kube-proxy EKS managed addon have configurable limits and requests for different instance types

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
What do you want us to build?

For EKS with mixed instances, we would like a way in which we could have two configurations for kube-proxy with different limits and requests so as to accommodate the different instance types.

Please help by building this for us

**Which service(s) is this request for?**

EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

We operate heterogeneous EKS clusters with mixed node types - both small nodes (e.g., t3.medium with 2 vCPU/4GB RAM) and large nodes (e.g., m5.4xlarge with 16 vCPU/64GB RAM) in the same cluster.

We need to optimize resource allocation for system components like kube-proxy to match the capacity of each node type.

Why it's hard:

Currently, EKS managed add-ons only support a single configuration per cluster. This means we must choose between:

1. Over-provisioning small nodes: Setting high resource requests/limits that work for large nodes but waste resources on small nodes

2. Under-provisioning large nodes: Setting conservative limits that fit small nodes but don't utilize the capacity of large nodes

3. Staying with self-managed components: Losing the operational benefits of managed add-ons

Current workaround limitations:

We currently run two self-managed kube-proxy DaemonSets with node selectors:

- kube-proxy-large with higher resource requests (200m CPU, 256Mi memory)
- kube-proxy-small with lower resource requests (50m CPU, 64Mi memory)

This works but requires us to maintain our own:

- Configuration management
- Version compatibility testing
- Operational monitoring

Desired Outcome

What we want to achieve:

- Use EKS managed add-ons for operational benefits (automatic updates, AWS support, security patching)
- Maintain optimal resource utilization across heterogeneous node types
- Eliminate the operational overhead of self-managed system components

Proposed solution:

Allow EKS managed add-ons to support node-group-specific or node-selector-based configurations, such as:

configurations:
- nodeSelector:
node.kubernetes.io/instance-type: "t3.medium"
resources:
requests: { cpu: "50m", memory: "64Mi" }
limits: { cpu: "100m", memory: "128Mi" }
- nodeSelector:
node.kubernetes.io/instance-type: "m5.4xlarge"
resources:
requests: { cpu: "200m", memory: "256Mi" }
limits: { cpu: "400m", memory: "512Mi" }

Business Impact
==========

Cost implications:

- Small nodes: Over-provisioning wastes ~150m CPU and ~192Mi memory per node
- Scale impact: In a 100-node cluster with 70% small nodes, this represents ~10.5 vCPU cores of wasted allocation
- Large nodes: Under-provisioning may lead to performance issues during high network load

Operational impact:

- Security risk: Self-managed components require manual security patching
- Operational overhead: Maintaining separate DaemonSets, monitoring, and updates
- Consistency: Mixed managed/self-managed architecture increases complexity

Industry pattern:

This is a common pattern in cost-optimized, multi-workload clusters where different node types serve different purposes (batch jobs on large nodes, web services on small nodes).

Technical Precedent
=============

This pattern already exists in Kubernetes ecosystem:

- Multiple DaemonSets with node selectors is a standard practice
- Other managed services (GKE Autopilot) handle this automatically
- The underlying Kubernetes platform supports this natively

Summary: We need EKS managed add-ons to support the same resource optimization patterns that self-managed Kubernetes components already enable, without forcing us to choose between operational simplicity and resource efficiency.

**Are you currently working around this issue?**

Still using self-managed kube-proxy at the moment with no workaround for EKS managed addon

**Additional context**
Anything else we should know?

**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

No repository files, tests, or entry points are named. Start by reviewing the proposed nodeSelector-based configurations and the current EKS managed add-on limitations described in the issue. Done means defining how different instance types receive kube-proxy resource requests and limits while retaining managed add-on operations.

Written by the indexing model from the issue text.