I think this comment is misleading
- Dominant language
- C++
- Stars
- 3.5k
- Forks
- 255
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/google/robotstxt/blob/750aec7933648c816d6d5bb2f4fe5c30f2485ccf/robots.h#L114
It says returns true iff *any* user agent in the vector is allowed to crawl. In fact, what it appears to me that it does is effectively collapse all rules that apply to any of the user agents in the vector into a single ruleset and then evaluate against that. That isn't always the same as any in the list being allowed.
e.g.
robots.txt:
```
User-agent: googlebot
Disallow: /foo/
```
if we call this method against the url `/foo/` with a vector containing both `googlebot` and `otherbot`, it will return `FALSE` even though clearly `otherbot` is allowed to crawl `/foo/` because (as I understand it) it's doing the equivalent of finding all rules that apply to either ua and collapsing into a single ruleset like:
```
User-agent: googlebot
User-agent: otherbot
Disallow: /foo/
```
So I think the comment is misleading, but would appreciate more eyes on the question!
Contributor guide
Assessment
This issue has not been assessed yet.