elastic / elastic/crawler

Validate command should output a warning if seed URL domains don't match main domain

Open
#342 0 comments 0 reactions 0 assignees View on GitHub
enhancement team:extract-and-transform
Dominant language
Ruby
Stars
224
Forks
48
Avg merge
23h 16m
Merged PRs (30d)
18

Description

If you call `bin/crawler validate ` for a configuration like below, the response says it's a valid URL. That's because the domain is valid, however the seed URLs are invalid.

```yaml
domains:
- url: https://example.com
seed_urls:
- https://example2.com
```

The above configuration _will_ crawl, but only because the main domain is used as a fallback seed URL. There are no warnings or errors about this misconfiguration during the crawl.

Invalid seed URLs are simply discarded when building the initial seed URL array to begin the crawl. This discarding is silent (no logs), so as a user figuring out what is wrong with my seed URLs can be very confusing.

So potential improvements:

- [ ] `bin/crawler validate` should check seed URL validity
- [ ] Invalid seed URLs should be logged

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.