internetarchive / internetarchive/openlibrary
Add Bot-specific Nginx Rate Limit Rules
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Feature Request
### Problem / Opportunity
This issue proposes adding bot-specific Nginx rate limit rules to better manage and control automated traffic and protect Open Library resources. This will help reduce load and potential abuse from bots, especially those identified in our user agent deny checks (see PR #11276).
### Relevant PR
[#11276 Add meta-externalagent to user agent deny checks](https://github.com/internetarchive/openlibrary/pull/11276)
### Main Files To Update
- [docker/web_nginx.conf](https://github.com/internetarchive/openlibrary/blob/master/docker/web_nginx.conf)
- [docker/nginx.conf](https://github.com/internetarchive/openlibrary/blob/master/docker/nginx.conf)
### Breakdown
### Related files
- [docker/web_nginx.conf](https://github.com/internetarchive/openlibrary/blob/master/docker/web_nginx.conf)
- [docker/nginx.conf](https://github.com/internetarchive/openlibrary/blob/master/docker/nginx.conf)
Contributor guide
Research direction
Start with docker/web_nginx.conf and docker/nginx.conf, then review PR #11276 for the user-agent deny checks that identify relevant bots. Determine the appropriate bot-specific rate-limit rules and verify that both Nginx configurations apply them consistently. Done means automated traffic is rate-limited to protect Open Library resources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nginx
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100