internetarchive / internetarchive/heritrix3
Refetching of failed URLs based on HTTP status codes/content
Open
feature request
hacktoberfest
pull request welcome
- Dominant language
- Java
- Stars
- 3.3k
- Forks
- 793
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 8
Description
Heritrix should support failing URL refetch and automatic retry later if HTTP status code is user configured one (For Example CloudFlare uses special codes like 529 when content is fetched from a site too quickly) or if the returned content matches regular expression(s).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.