Too long sleep time on http400 error
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
I oversee the operation of over 200 onpremis machines with SSM
We implement frequent updates.
When I upgraded agent version from 3.1.1927 to 3.2.286, persistent disconnections emerged.
The accompanying log records instances of these occurrences.
> WARN [CredentialRefresher] Status code %!s(int=400) returned from AWS API. RequestId: c9f7f596-2b20-4b0c-bcee-XXXXXXXXXXXX Message: The connection was closed by the server at some point between the client sending the request and the client receiving the entire response
> INFO [CredentialRefresher] Sleeping for 25h17m57s before retrying retrieve credentials
>
This issue transpires on a daily basis on one of our servers.
I don't believe network malfunctions are the root cause, as restarting the SSM Agent Service resolves the disconnections.
A review of the source code reveals instructions to allow for a 24-hour wait for HTTP error code 400. This seems excessively prolonged.
A few additional attempts, among other remedial measures, could potentially eliminate such extended disconnection incidents.
Current version is 3.2.419.0 but it still keeps happening.
Contributor guide
Research direction
Start by locating the Go code that handles CredentialRefresher HTTP 400 responses and calculates the retry sleep, then compare it with the reported 25h17m57s log. The issue names no file or test; done means reproducing or covering this case and confirming that retries no longer impose an unintended 24-hour wait.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100