aws / aws/aws-advanced-go-wrapper
awsSecretsManager: configurable connect-level retry/backoff to bridge the secret rotation window
- Dominant language
- Go
- Stars
- 78
- Forks
- 12
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 13
Description
### Describe the feature
First of all, thank you for building and maintaining the AWS advanced go wrapper.
I'd like to raise one gap I ran into:
Add an opt-in, configurable **retry/backoff on the connect path** to the
`awsSecretsManager` plugin, so that it can keep retrying (forced refetch +
reconnect) across a Secrets Manager rotation window until `AWSCURRENT` is
promoted to the new secret version.
Today the plugin retries **at most once**, and only when the *cached* secret
failed (`aws_secrets_manager_connection_plugin.go`, `connectInternal`):
```go
secretsWasFetched, _ := updateSecrets(..., false) // use cache if present (no API call)
applySecretToProperties(props)
conn, err := connectFunc(props) // attempt #1
if err == nil { return conn }
if IsLoginError(err) && !secretsWasFetched { // only when cache failed
secretsWasFetched, err = updateSecrets(..., true) // force refetch (1 API call)
if secretsWasFetched {
applySecretToProperties(props)
return connectFunc(props) // attempt #2 — and that's it
}
}
return nil, err
```
I'd like the plugin to optionally accept a total time budget + capped
exponential backoff, polling `GetSecretValue` / reconnect until the rotation
window closes, instead of giving up after a single forced refetch.
### Use Case
I'm frustrated when a managed Secrets Manager rotation causes **every new
physical connection to fail for ~1 minute**.
As far as my understanding, rotation goes through `createSecret → setSecret → testSecret → finishSecret`.
Between `setSecret` (the DB password is already changed) and `finishSecret`
(`AWSCURRENT` promoted to the new version) there is a window where:
- the DB already expects the **new** password, but
- `GetSecretValue(AWSCURRENT)` still returns the **old** password.
In my environment (Aurora PostgreSQL, managed rotation) this window was
measured at **~1 minute** (e.g. `12:10:31 → 12:11:45`, ≈ 74s) from application
logs. Inside that window the plugin's single retry cannot help:
1. cached (old) secret → connect fails (`IsLoginError`),
2. forced refetch → `AWSCURRENT` is **still the old secret** (not yet promoted),
3. attempt with the same old secret → fails again,
4. plugin gives up and returns the login error (`SQLSTATE 28P01` / `28000`).
Existing pooled connections keep working, but every **new** physical connection
fails — startup (first connection), pool growth under load, and
`ConnMaxLifetime` recycle. The plugin only recovers **after** `finishSecret`.
### Proposed Solution
- Add a configurable connect-retry/backoff to the `awsSecretsManager` plugin
(e.g. `secretsManagerConnectRetryTimeoutMs` + capped exponential backoff) so it
can poll across the rotation window until `AWSCURRENT` is promoted; **or**
- If this is intentionally out of scope, document that the plugin is *not*
expected to bridge the rotation window and that callers must implement their
own connect-level retry; **or**
- Provide guidance on whether **RDS Proxy** is the
intended way to absorb rotation on the wrapper's behalf.
Concretely: **for the Secrets Manager plugin, what is the recommended way to
survive the rotation window — application-side connect retry, or RDS Proxy?**
### Other Information
_No response_
### Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### The AWS Advanced Go Wrapper version used
v1.1.0
### Go version used
go1.25.5
### Operating System and version
Runtime: AWS Lambda (Amazon Linux 2, x86_64)
Contributor guide
Research direction
Start by reading aws_secrets_manager_connection_plugin.go and its connectInternal entry point, then trace updateSecrets and connectFunc. Done should be an agreed opt-in connect-level retry budget with capped backoff that refetches and reconnects through the rotation window, or documentation explaining the supported alternative if this remains out of scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, postgresql
- Domain
- backend, cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100