adobe / adobe/spacecat-audit-worker

Use stored scraped content for top pages instead of crawling inside canonical audit

Open
#355 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
14
Forks
15
Avg merge
1d 21m
Merged PRs (30d)
58

Description

**Is your feature request related to a problem? Please describe.**
https://github.com/adobe/spacecat-audit-worker/pull/326/files#diff-88d6eda336d671388349471e2f7210bbb88fba4bd0d4fdbbd7b02cc632aaadc7R147
instead of fetching all the URLs here, we could leverage the scraped content stored in S3, so we don't need to make a request for each audit again (canoincal, JSON-LD, hreflang...)
https://wiki.corp.adobe.com/display/AEMSites/SpaceCat+%7C+Core+%7C+Content+Scraper+and+Processing

**Describe the solution you'd like**
Use stored scraped content for top pages instead of crawling inside canonical audit

**Additional context**
[Add any other context or screenshots about the feature request here.
](https://github.com/adobe/spacecat-audit-worker/pull/326)

Contributor guide

Open the contributing guide

Research direction

Start with the canonical audit URL-fetching logic referenced in PR #326 and review the SpaceCat Content Scraper and Processing documentation. Trace how scraped content is stored in S3 and how the canonical audit currently fetches top-page URLs; done means the audit uses the stored content instead of making those repeated requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.