Enable CloudFront rate limiting for Catalog and Harvest
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 4h 56m
- Merged PRs (30d)
- 5
Description
[Slack discussion](https://gsa-tts.slack.com/archives/C2N85536E/p1788292223614329)
We start to see individual client IPs generating sustained high request volumes against catalog.data.gov. We should enable rate-limit rules on CloudFront/WAF and apply them to catalog and harvest
## Sketch
Do not expose sensitive information.
- determine the request patterns the rate-limit rules should apply to. Discuss and finalize in Slack.
- determine an appropriate rate limit (e.g., ### requests per 5 minutes per IP).
- allow matching traffic, but add a header to flagged requests so it can be captured and logged by Nginx.
- document the rule in cloudfront docs in google drive([catalog cloudfront](https://docs.google.com/document/d/1Ieo39Ka3BdkGq7xPYMq7lTU44Y0Fz3Zdo5lo-207Sec/edit?tab=t.0) and [harvest cloudfront](https://docs.google.com/document/d/1Hz5h3xWAqwKJv-rCM0U3opHnxJklSXsGE_j6L8FOnQE/edit?tab=t.0)), and send the requests to cg support of have them deployed.
- build New Relic dashboards to monitor the captured traffic
Contributor guide
Assessment
This issue has not been assessed yet.