google / google/robotstxt

An encoding test does not appear to match the RFC?

Open
#64 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.5k
Forks
255
PR merge metrics
No merged PRs in 30d

Description

The [first `ID_Encoding` test](https://github.com/google/robotstxt/blob/455b1583103d13ad88fe526bc058d6b9f3309215/robots_test.cc#L399) caught me by surprise, since it does not _appear_ to match the RFC:

```c++
// /foo/bar?baz=http://foo.bar stays unencoded.
{
const absl::string_view robotstxt =
"User-agent: FooBot\n"
"Disallow: /\n"
"Allow: /foo/bar?qux=taz&baz=http://foo.bar?tar&par\n";
EXPECT_TRUE(IsUserAgentAllowed(
robotstxt, "FooBot",
"http://foo.bar/foo/bar?qux=taz&baz=http://foo.bar?tar&par"));
}
```

However, [section 2.2.2 of the REP RFC](https://datatracker.ietf.org/doc/html/rfc9309#section-2.2.2) seems to indicate that `/foo/bar?baz=http://foo.bar` should be encoded as `/foo/bar?baz=http%3A%2F%2Ffoo.bar`.

I can't decide if I'm mis-reading the RFC or if the test intentionally deviates from the RFC in this case.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.