Delay between requests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.1k
- Forks
- 355
- PR merge metrics
- No merged PRs in 30d
Description
Where a URL list contains many pages on the same site would it be possible to add an arbitrary delay (possibly random) between consecutive requests? Our own web page has a software firewall that prevents access from addresses making more than, for example, 20 requests per minute. If my watch process takes 10 minutes rather than 10 seconds that is not an issue, as long as it checks all the pages and doesn't get blocked. Something like a
PauseBetweenRequests : 10 seconds
in the config file would allow checking of sites that have countermeasures for scripted requests.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests. Start by locating the configuration parsing and the loop that makes consecutive requests, then trace how a delay setting could apply between requests to the same site. Done means the configured delay is honored, including the requested fixed or random behavior, without preventing all pages from being checked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100