internetarchive / internetarchive/openlibrary

Add Bot-specific Nginx Rate Limit Rules

Open
#11,277 2 comments 0 reactions 0 assignees View on GitHub
Affects: Operations Lead: @scottbarnes Needs: Staff / Internal Priority: 2 Theme: Performance
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Feature Request

### Problem / Opportunity
This issue proposes adding bot-specific Nginx rate limit rules to better manage and control automated traffic and protect Open Library resources. This will help reduce load and potential abuse from bots, especially those identified in our user agent deny checks (see PR #11276).

### Relevant PR
[#11276 Add meta-externalagent to user agent deny checks](https://github.com/internetarchive/openlibrary/pull/11276)

### Main Files To Update
- [docker/web_nginx.conf](https://github.com/internetarchive/openlibrary/blob/master/docker/web_nginx.conf)
- [docker/nginx.conf](https://github.com/internetarchive/openlibrary/blob/master/docker/nginx.conf)

### Breakdown

### Related files
- [docker/web_nginx.conf](https://github.com/internetarchive/openlibrary/blob/master/docker/web_nginx.conf)
- [docker/nginx.conf](https://github.com/internetarchive/openlibrary/blob/master/docker/nginx.conf)

Contributor guide

Open the contributing guide

Research direction

Start with docker/web_nginx.conf and docker/nginx.conf, then review PR #11276 for the user-agent deny checks that identify relevant bots. Determine the appropriate bot-specific rate-limit rules and verify that both Nginx configurations apply them consistently. Done means automated traffic is rate-limited to protect Open Library resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
nginx
Domain
infrastructure
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.