[ECR] [request]: CloudWatch metrics for ECR Replication lag
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
I'd like to have metrics for visualizing ECR replication lags for different images in different regions.
**Which service(s) is this request for?**
ECR
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
As far as I know, there is currently no SLA regarding ECR replication time, and I didn't find anything in the documentation that clarifies what we should expect.
While most replications seem to be fairly fast (a few seconds), we've encountered occasional issues where replication took longer, especially when replicating from us to eu (e.g.: us-west-2 replicated to eu-central-1/eu-west-1). However, we don't have a good way to visualize what is a typical replication time for our images, whether we encounter replication lag spikes (if so, how frequently? how long? in which regions/images?), or even configure alerting for replication lag issues.
We were excited when https://github.com/aws/containers-roadmap/issues/1193 shipped! While it helps us to hold off deployments in some regions until the ECR API tells us the image has been replicated, it does not improve much our ability to monitor replication lags across images/regions.
**Are you currently working around this issue?**
There does not seem to be a good workaround for us to monitor things.
I guess one way would be to call `aws ecr describe-images` for each region, since this API returns `imagePushedAt` (where `imagePushedAt` seem to contain the time when the image was replicated). We could call this API when `aws ecr describe-image-replication-status` is letting us know that an image was replicated after pushing it. We could then determine the replication like by comparing the `imagePushedAt` between the source & destination regions and publish that somewhere (Datadog?).
**Additional context**
N/A
**Attachments**
N/A
Contributor guide
Research direction
Start by reviewing the ECR replication status and image APIs mentioned in the issue: describe-image-replication-status and describe-images, including imagePushedAt across source and destination regions. Define the metrics needed to visualize replication lag by image and region and support alerting. Done should provide a documented way to measure and monitor replication lag, with behavior that can be validated against those APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100