Allow optional ingestion of sitemap data
Open
complexity:low
enhancement
priority:low
- Dominant language
- Ruby
- Stars
- 224
- Forks
- 48
- Avg merge
- 23h 16m
- Merged PRs (30d)
- 18
Description
### Problem Description
Crawler uses sitemaps to seed URLs for a crawl, but the content of sitemap files is never ingested. Some users may want to ingest data from this (for example metadata).
### Proposed Solution
- Add a config option to allow ingesting sitemap content
- If config is enabled, ingest content of sitemap after using it to seed more URLs
Contributor guide
Assessment
This issue has not been assessed yet.