GSA / GSA/data.gov

Enable CloudFront rate limiting for Catalog and Harvest

Open
#6,284 2 comments 0 reactions 1 assignee Claimed by @Aguiardavidm View on GitHub
bug
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

[Slack discussion](https://gsa-tts.slack.com/archives/C2N85536E/p1788292223614329)

We start to see individual client IPs generating sustained high request volumes against catalog.data.gov. We should enable rate-limit rules on CloudFront/WAF and apply them to catalog and harvest

## Sketch
Do not expose sensitive information.

- determine the request patterns the rate-limit rules should apply to. Discuss and finalize in Slack.
- determine an appropriate rate limit (e.g., ### requests per 5 minutes per IP).
- allow matching traffic, but add a header to flagged requests so it can be captured and logged by Nginx.
- document the rule in cloudfront docs in google drive([catalog cloudfront](https://docs.google.com/document/d/1Ieo39Ka3BdkGq7xPYMq7lTU44Y0Fz3Zdo5lo-207Sec/edit?tab=t.0) and [harvest cloudfront](https://docs.google.com/document/d/1Hz5h3xWAqwKJv-rCM0U3opHnxJklSXsGE_j6L8FOnQE/edit?tab=t.0)), and send the requests to cg support of have them deployed.
- build New Relic dashboards to monitor the captured traffic

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.