addyosmani / addyosmani/agentic-seo
Error: No AI crawlers are explicitly allowed in robots.txt
Offen
- Vorherrschende Sprache
- JavaScript
- Sterne
- 309
- Forks
- 50
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
Repro:
agentic-seo --url https://docs.nvidia.com/dynamo/getting-started/quickstart
OR
agentic-seo --url https://docs.nvidia.com/dynamo/getting-started/quickstart.md (ran sequentially)
Warning returned:
△ [Robots.txt AI Access] No AI crawlers are explicitly allowed in robots.txt.
There's a robots.txt at https://docs.nvidia.com/robots.txt and it does explicitly allow AI crawlers (not that it should be needed as robots is an **exclusion protocol,** not a permissions protocol).
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.