argoproj / argoproj/argo-workflows

use internal exponential backoff to avoid flapping on DB startup

Open
#8,797 1 comment 1 reaction 0 assignees View on GitHub
area/workflow-archive solution/suggested
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

Walking through the quickstart, I notice that both the argo-server and workflow controller "flap" while waiting for postgres to become available. On average (on my machine, at least), both components restart 3 times before coming up clean. This is by no means an out of the ordinary thing for k8s apps, however, if either of those components get too far into a crashloop backoff, the overall effect can be that the system _as a whole_ takes longer than it ought to to come up clean.

I wanted to propose that it's fairly easy to implement an exponential backoff (with a low max backoff between retries) _internally_ so that components don't "flap" like this while waiting for their own network-bound dependencies to be satisfied. Speaking [from experience](https://github.com/brigadecore/brigade/pull/1827/files), this strategy can allow a system such as this one to start faster and smoother, as a whole.

---

**Message from the maintainers**:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

Contributor guide

Open the contributing guide

Research direction

Start by tracing startup and dependency-retry behavior in the argo-server and workflow controller while following the quickstart against an unavailable PostgreSQL service. Done means both components use bounded internal exponential backoff and the quickstart no longer shows repeated restarts before PostgreSQL becomes available.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
backend, databases, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.