Add option to not crawl URLs already crawled in an index
- Dominant language
- Ruby
- Stars
- 224
- Forks
- 48
- Avg merge
- 23h 16m
- Merged PRs (30d)
- 18
Description
### Problem Description
I think it would be valuable to have an option to avoid duplicate crawls across runs. E.g., check an index to see if the given url has already been crawled - if so, don't crawl it again.
### Proposed Solution
Something to the effect of:
```bash
bin/crawler crawl config/cisa_cybersecurity_advisories.yaml \
--es-config config/elasticsearch.yaml
--no-duplicates-index
```
Now the above should check the `` index for URLs that have been crawled already.
### Alternatives
Maybe I could get a list of all the URLs that have been crawled and pass them into the crawl rules (specified in `CRAWL_RULES.md`) as _disallowed crawls_.
### Additional Context
N/A
Contributor guide
Assessment
This issue has not been assessed yet.