addyosmani / addyosmani/agentic-seo

Error: No AI crawlers are explicitly allowed in robots.txt

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
309
Forks
50
PR merge metrics
No merged PRs in 30d

Description

Repro:
agentic-seo --url https://docs.nvidia.com/dynamo/getting-started/quickstart
OR
agentic-seo --url https://docs.nvidia.com/dynamo/getting-started/quickstart.md (ran sequentially)

Warning returned:
△ [Robots.txt AI Access] No AI crawlers are explicitly allowed in robots.txt.

There's a robots.txt at https://docs.nvidia.com/robots.txt and it does explicitly allow AI crawlers (not that it should be needed as robots is an **exclusion protocol,** not a permissions protocol).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.