ETCD throttling on EKS

Summary

Argo-Workflows produces a lot of API calls to ETCD and some of the requests can be canceled and Workflow transition between states can be disrupted. You can spot a error on the controller:
cannot validate Workflow: rpc error: code = ResourceExhausted desc = etcdserver: throttle: too many requests

Use Cases

There is a self-calculated throttling limits on AWS EKS and if your K8s cluster is small, ETCD can throttle some updates of Workflows. For example, we have many Cronworkflows and ETCD can skip the update from Running state of Workflow to Finished and it gets stuck forever because the controller is waiting when Workflow is going to be finished. Pods of such Workflow are gone when they are completed, and no sidecontainers will try to update the status again.

Probably, some mechanism of retries should be implemented to avoid this and have a guarantee that the state of Workflow will be changed.

For AWS EKS there is a workaround: scale you cluster up for a short period of time to boost the throttling limit.

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

Start with the workflow controller path associated with the reported cannot validate Workflow error and trace how a Workflow moves from Running to Finished. Define the retry behavior for ResourceExhausted/throttled updates and verify that completed Workflows cannot remain stuck when their pods are gone.