google / google/robotstxt

I think this comment is misleading

Open
#27 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.5k
Forks
255
PR merge metrics
No merged PRs in 30d

Description

https://github.com/google/robotstxt/blob/750aec7933648c816d6d5bb2f4fe5c30f2485ccf/robots.h#L114

It says returns true iff *any* user agent in the vector is allowed to crawl. In fact, what it appears to me that it does is effectively collapse all rules that apply to any of the user agents in the vector into a single ruleset and then evaluate against that. That isn't always the same as any in the list being allowed.

e.g.

robots.txt:

```
User-agent: googlebot
Disallow: /foo/
```

if we call this method against the url `/foo/` with a vector containing both `googlebot` and `otherbot`, it will return `FALSE` even though clearly `otherbot` is allowed to crawl `/foo/` because (as I understand it) it's doing the equivalent of finding all rules that apply to either ua and collapsing into a single ruleset like:

```
User-agent: googlebot
User-agent: otherbot
Disallow: /foo/
```

So I think the comment is misleading, but would appreciate more eyes on the question!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.