argoproj / argoproj/argo-workflows
use internal exponential backoff to avoid flapping on DB startup
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
Walking through the quickstart, I notice that both the argo-server and workflow controller "flap" while waiting for postgres to become available. On average (on my machine, at least), both components restart 3 times before coming up clean. This is by no means an out of the ordinary thing for k8s apps, however, if either of those components get too far into a crashloop backoff, the overall effect can be that the system _as a whole_ takes longer than it ought to to come up clean.
I wanted to propose that it's fairly easy to implement an exponential backoff (with a low max backoff between retries) _internally_ so that components don't "flap" like this while waiting for their own network-bound dependencies to be satisfied. Speaking [from experience](https://github.com/brigadecore/brigade/pull/1827/files), this strategy can allow a system such as this one to start faster and smoother, as a whole.
---
**Message from the maintainers**:
Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.
Contributor guide
Research direction
Start by tracing startup and dependency-retry behavior in the argo-server and workflow controller while following the quickstart against an unavailable PostgreSQL service. Done means both components use bounded internal exponential backoff and the quickstart no longer shows repeated restarts before PostgreSQL becomes available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, postgresql
- Domain
- backend, databases, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100