adobe / adobe/spacecat-audit-worker
Use stored scraped content for top pages instead of crawling inside canonical audit
- Dominant language
- JavaScript
- Stars
- 14
- Forks
- 15
- Avg merge
- 1d 21m
- Merged PRs (30d)
- 58
Description
**Is your feature request related to a problem? Please describe.**
https://github.com/adobe/spacecat-audit-worker/pull/326/files#diff-88d6eda336d671388349471e2f7210bbb88fba4bd0d4fdbbd7b02cc632aaadc7R147
instead of fetching all the URLs here, we could leverage the scraped content stored in S3, so we don't need to make a request for each audit again (canoincal, JSON-LD, hreflang...)
https://wiki.corp.adobe.com/display/AEMSites/SpaceCat+%7C+Core+%7C+Content+Scraper+and+Processing
**Describe the solution you'd like**
Use stored scraped content for top pages instead of crawling inside canonical audit
**Additional context**
[Add any other context or screenshots about the feature request here.
](https://github.com/adobe/spacecat-audit-worker/pull/326)
Contributor guide
Research direction
Start with the canonical audit URL-fetching logic referenced in PR #326 and review the SpaceCat Content Scraper and Processing documentation. Trace how scraped content is stored in S3 and how the canonical audit currently fetches top-page URLs; done means the audit uses the stored content instead of making those repeated requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, javascript
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100