adobe / adobe/spacecat-shared

composeAuditURL does not return the correct URL in some cases of domains with bot protection

Open
#237 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
5
Forks
2
Avg merge
19h 55m
Merged PRs (30d)
33

Description

**Description**
composeAuditURL is used in both audit and import workers to return the URL to audit/import data for.
composeAuditURL does not return the correct URL in some cases of domains with bot protection, see example in slack thread below.

An empty response from RUM/ahrefs APIs doesn't always mean we're querying for the wrong URL. e.g. broken backlinks is expected sometimes to return empty, sometimes with broken backlinks. So we cannot just assume empty response => fallback to www, like the fix implemented in https://github.com/adobe/spacecat-audit-worker/pull/248.

**Expected behavior**
composeAuditURL or a similar shared function should return a correct URL for those cases as well, and audits/imports should be updated to that version.

**Additional context**
See https://cq-dev.slack.com/archives/C05A45JBP9N/p1715068794711109

composeAuditURL: https://github.com/adobe/spacecat-shared/blob/9e0dc2cdc080411597f47319b5dcc2bedf764ada/packages/spacecat-shared-utils/src/url-helpers.js#L86

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.