dice-group / dice-group/Squirrel
Worker should follow instructions in HTTP 429 responses
Open
component:worker
priority:low
type:enhancement
- Dominant language
- Java
- Stars
- 25
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
### Description
As a website owner, I expect crawlers to behave. If they are crawling too fast _and_ my server responds with [HTTP 429 (RFC 6585)](https://tools.ietf.org/html/rfc6585#page-3), the crawler should follow the instructions and slow down.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the crawler worker's HTTP response handling and any existing crawl-delay logic, then read the linked HTTP 429/RFC 6585 guidance. Done means the crawler follows the server's instructions and slows down after a 429 response, with coverage for that behavior if the project has relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, networking
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100