elastic / elastic/crawler

Add option to not crawl URLs already crawled in an index

Open
#105 2 comments 1 reaction 0 assignees View on GitHub
community-driven complexity:high enhancement priority:low
Dominant language
Ruby
Stars
224
Forks
48
Avg merge
23h 16m
Merged PRs (30d)
18

Description

### Problem Description

I think it would be valuable to have an option to avoid duplicate crawls across runs. E.g., check an index to see if the given url has already been crawled - if so, don't crawl it again.

### Proposed Solution

Something to the effect of:

```bash
bin/crawler crawl config/cisa_cybersecurity_advisories.yaml \
--es-config config/elasticsearch.yaml
--no-duplicates-index
```

Now the above should check the `` index for URLs that have been crawled already.

### Alternatives
Maybe I could get a list of all the URLs that have been crawled and pass them into the crawl rules (specified in `CRAWL_RULES.md`) as _disallowed crawls_.

### Additional Context

N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.