[EKS] [request]: Enable Taints and Tolerations on AWS FIS Pods
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
What do you want us to build?
Currently, fault injection pods created by AWS FIS experiment templates are fully managed by the FIS service. Unlike native EKS workloads, where users can define tolerations in the pod specification to accommodate node taints, FIS does not provide any such configuration option. This prevents FIS pods from being scheduled on clusters where all worker nodes are configured with custom taints.
When an AWS FIS experiment action is initiated, FIS retrieves the FIS Pod container image. This image is then used to create a Pod in the targeted EKS cluster. The newly-created Pod is responsible for injecting, controlling, and monitoring the fault. However, FIS does not currently provide a way to specify pod template specifications like tolerations or node affinity for these helper pods.
For some customer environments, due to other ongoing priorities it may not be possible to have untainted nodes. So, the request is to enable support for native Kubernetes capabilities like tolerations.
**Which service(s) is this request for?**
AWS Fault Injection Service (FIS), Amazon EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We operate a multi-region EKS platform (NXOP) running mission-critical airline operations workloads. All worker nodes in our clusters are configured with custom taints (e.g., dedicated=nxop:NoSchedule) to enforce strict workload isolation and prevent unauthorized pods from consuming compute resources.
When we attempt to run FIS experiments using aws:eks:pod-* actions (such as aws:eks:pod-network-latency, aws:eks:pod-cpu-stress, etc.), the FIS-managed fault injection pods fail to schedule because they lack the necessary tolerations to match our node taints. The pods remain in Pending state indefinitely, and the
experiment times out.
The FIS Pod action parameters only expose: fisPodLabels, fisPodAnnotations, fisPodContainerImage, and fisPodSecurityPolicy. Labels and annotations have no effect on Kubernetes scheduling decisions — only tolerations and nodeSelector/nodeAffinity do, and none of these are configurable.
This makes it impossible to use FIS EKS pod-level fault injection in any cluster with a taint-based isolation strategy, which is a common security and operational best practice in enterprise environments.
The impact is significant: we cannot validate our DR failover procedures, chaos engineering practices, or application resilience under fault conditions using the native AWS tooling. This forces us to either maintain untainted nodes (which breaks our isolation model) or abandon FIS entirely for pod-level experiments.
**Are you currently working around this issue?**
There is no viable workaround that preserves our security posture:
1. Adding untainted nodes — Not feasible. Our cluster node groups are uniformly tainted by policy, and maintaining a separate untainted node group solely for FIS pods introduces security risk, operational overhead, and cost.
2. Temporarily removing taints — Requires manual intervention before each experiment, creates a window where unintended workloads could schedule, and defeats the purpose of automated chaos testing.
3. Using non-FIS tools (e.g., Litmus, Chaos Mesh) — These support tolerations natively, but we lose the AWS-native integration with FIS (experiment templates, IAM controls, CloudWatch integration, stop conditions, guardrails).
None of these are acceptable for production use.
**Additional context**
Requested enhancement: Add optional fisPodTolerations and fisPodNodeSelector (or fisPodNodeAffinity) parameters to aws:eks:pod-* actions, following the same pattern as the existing fisPodLabels and fisPodAnnotations parameters.
Example desired configuration:
```json
{
"fisPodTolerations": [
{
"key": "dedicated",
"operator": "Equal",
"value": "nxop",
"effect": "NoSchedule"
}
]
}
```
This would be a backward-compatible addition — existing experiments without tolerations would continue to work as-is. Clusters without taints are unaffected.
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
Contributor guide
Research direction
No repository files, tests, or entry points are named. Start by checking the AWS FIS EKS pod-* action documentation and parameter support, then verify how Kubernetes tolerations and node selectors are represented. Done would mean optional toleration and placement settings are supported for FIS-managed pods without affecting existing experiments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100