`is_token` implementation is more permissive than standard

RFC 9110 Section 5.6.2 defines the grammar for field value tokens as

  token          = 1*tchar

  tchar          = "!" / "#" / "$" / "%" / "&" / "'" / "*"
                 / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
                 / DIGIT / ALPHA
                 ; any VCHAR, except delimiters

However in practice it seems that this specific grammar is relaxed to a larger set of allowed characters. I made a small test which checks this behavior:

#[test]
fn test_header_with_invalid_rfc91100_token_in_value() {
    const RESPONSE: &[u8] =
        b"HTTP/1.1 200 OK\r\nAccess-Control-Allow-Credentials: hell[o\r\nBread: baguette\r\n\r\n";

    let mut headers = [EMPTY_HEADER; 2];
    let mut response = Response::new(&mut headers[..]);

    let result = crate::ParserConfig::default().parse_response(&mut response, RESPONSE);
    let status = result.unwrap();

    assert_eq!(status, Status::Complete(78));
    assert_eq!(response.version.unwrap(), 1);
    assert_eq!(response.code.unwrap(), 200);
    assert_eq!(response.reason.unwrap(), "OK");
    assert_eq!(response.headers.len(), 2);
    assert_eq!(response.headers[0].name, "Access-Control-Allow-Credentials");
    assert_eq!(response.headers[0].value, &b"hell[o"[..]);
    assert_eq!(response.headers[1].name, "Bread");
    assert_eq!(response.headers[1].value, &b"baguette"[..]);
}

I also checked how Firefox handles this header with a small program

printf "HTTP/1.1 200 OK\r\nAccess-Control-Allow-Credentials: hell[o\r\nBread: baguette\r\n\r\n" | nc -l 4040

Then visited localhost:4040 in Firefox. The recorded execution also showed the delimiter in the token value:

Based on the test assertions and the firefox screenshot it seems consistent that both programs allow characters outside the grammar specified in RFC91100.

• Was this intentional at the time of writing the tokenization code?
• Would it possible to have a strict mode which would reject more field values for invalid tokens?

Start by locating the is_token implementation and the ParserConfig::parse_response entry point. Reproduce the behavior with the provided test_header_with_invalid_rfc91100_token_in_value case, then determine the intended strict-mode behavior from the RFC and existing parser tests. Done should include a decided handling for invalid token characters and coverage for that behavior.