Argo CD stuck indefinitely while refreshing an App
- Dominant language
- Go
- Stars
- 24.2k
- Forks
- 7.8k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 234
Description
Checklist:
* [ Y] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
* [ Y] I've included steps to reproduce the bug.
* [ Y] I've pasted the output of `argocd version`.
**Describe the bug**
We are getting issues on multiple EKS clusters where Apps get stuck in refreshing state indefinitely.
**To Reproduce**
1. Enable Auto Sync.
2. Deploy a new App.
**Expected behavior**
App will get stuck in refreshing state indefinitely.
Even when we try to use command argocd app terminate-op APP-NAME, it doesn't help.
**Screenshots**
**Version**
v2.5.1
**Logs**
In logs we found that it times out after 3 mins, then again it retries to refresh the app status and again fails. This keeps on happening in loop infinitely.:
time="2023-09-15T08:04:19Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:04:04Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:03:19Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:03:04Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:04:34Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:05:05Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:05:20Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:05:35Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:06:05Z" level=info msg="Refreshing app status (comparison expired, requesting refresh. reconciledAt: 2023-09-15 08:03:04 +0000 UTC, expiry: 3m0s), level (2)" application=argocd/XXXX
Contributor guide
Research direction
Start by reproducing the issue with Auto Sync enabled and a newly deployed App, then inspect the controller logs around the three-minute refresh timeout and retry loop. Use `argocd app terminate-op APP-NAME` during reproduction. Done means the App can leave the refreshing state and refresh operations no longer loop indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- ci-cd, cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100