guardian / guardian/dotcom-rendering

Investigate spike in 5xx errors from router-frontend-PROD

Open
#14,014 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
274
Forks
34
Avg merge
2d 22h
Merged PRs (30d)
121

Description

Over the last couple of days we've seen a number of [spikes in 5xx errors from the router-frontend-PROD service](https://metrics.gutools.co.uk/d/zM0ouzs4z/availability-dashboard?orgId=1&var-Period=5m&var-AWSAccount=000000101&var-StageFilter=PROD&var-ElkIndexFilter=logstash-frontend-*&var-ElasticSearchIndex=W7i1CLSVk&var-Service=frontend-PROD-router-ELB&var-App=router&var-Stack=frontend&var-Stage=PROD&var-ASGName=frontend-PROD-RouterStack-G1EEL7AIYWIU-AutoscalingGroup-1DO4VJI0GY7JK&var-SLO=0.995&var-RiffRaffProject=dotcom%3Arouter&var-LogLevel=ERROR&var-KibanaErrorsURL=https%3A%2F%2Flogs.gutools.co.uk%2Fs%2Fdotcom%2Fapp%2Fdiscover%23%2F%3F_g%3D%28filters%3A%21%28%28query%3A%28match_phrase%3A%28app%3Arouter%29%29%29%2C%28query%3A%28match_phrase%3A%28stage%3APROD%29%29%29%2C%28query%3A%28match_phrase%3A%28level%3AERROR%29%29%29%29%2CrefreshInterval%3A%28pause%3A%21t%2Cvalue%3A0%29%2Ctime%3A%28from%3A%272025-05-28T09%3A46%3A50.890Z%27%2Cto%3A%272025-05-29T09%3A46%3A50.890Z%27%29%29&var-KibanaWarnURL=https%3A%2F%2Flogs.gutools.co.uk%2Fs%2Fdotcom%2Fapp%2Fdiscover%23%2F%3F_g%3D%28filters%3A%21%28%28query%3A%28match_phrase%3A%28app%3Arouter%29%29%29%2C%28query%3A%28match_phrase%3A%28stage%3APROD%29%29%29%2C%28query%3A%28match_phrase%3A%28level%3AWARN%29%29%29%29%2CrefreshInterval%3A%28pause%3A%21t%2Cvalue%3A0%29%2Ctime%3A%28from%3A%272025-05-28T09%3A46%3A50.890Z%27%2Cto%3A%272025-05-29T09%3A46%3A50.890Z%27%29%29&var-KibanaInfoURL=https%3A%2F%2Flogs.gutools.co.uk%2Fs%2Fdotcom%2Fapp%2Fdiscover%23%2F%3F_g%3D%28filters%3A%21%28%28query%3A%28match_phrase%3A%28app%3Arouter%29%29%29%2C%28query%3A%28match_phrase%3A%28stage%3APROD%29%29%29%2C%28query%3A%28match_phrase%3A%28level%3AINFO%29%29%29%29%2CrefreshInterval%3A%28pause%3A%21t%2Cvalue%3A0%29%2Ctime%3A%28from%3A%272025-05-28T09%3A46%3A50.890Z%27%2Cto%3A%272025-05-29T09%3A46%3A50.890Z%27%29%29&var-KibanaAppFilter=All&var-ASGNameFilter=%28%22AutoScalingGroupName%22%3D%22frontend-PROD-RouterStack-G1EEL7AIYWIU-AutoscalingGroup-1DO4VJI0GY7JK%22%29&from=1747915166307&to=1748519966307). In some cases these coincide with an overall increase in traffic. We should investigate to determine the underlying cause and whether we need to take any action to mitigate further errors.

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.