addyosmani / addyosmani/agentic-seo

Error: No AI crawlers are explicitly allowed in robots.txt

未关闭
#5 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
JavaScript
星标
309
派生
50
PR 合并指标
30 天内没有已合并 PR

描述

Repro:
agentic-seo --url https://docs.nvidia.com/dynamo/getting-started/quickstart
OR
agentic-seo --url https://docs.nvidia.com/dynamo/getting-started/quickstart.md (ran sequentially)

Warning returned:
△ [Robots.txt AI Access] No AI crawlers are explicitly allowed in robots.txt.

There's a robots.txt at https://docs.nvidia.com/robots.txt and it does explicitly allow AI crawlers (not that it should be needed as robots is an **exclusion protocol,** not a permissions protocol).

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。