Make PatternToken public

glob provides nice parsing and matching facilities, and sometimes it's important to reach into the Pattern and inspect the individual PatternToken elements themselves.

Use case

Given some fn extend_with_literals(input: &str, pattern: Pattern), identify whether the input is a valid prefix of pattern and determine any subsequent literal characters from the pattern that would better qualify the input.

Example

assert_eq!(
    extend_with_literals("pr", Pattern::new("pr[e]?ix").unwrap()),
    "pre"
);

assert_eq!(
    extend_with_literals("pre", Pattern::new("pr[e]?ix").unwrap()),
    "pre"
);

assert_eq!(
    extend_with_literals("pref", Pattern::new("pr[e]?ix").unwrap()),
    "prefix"
);

This makes it possible to optimize glob in over a nonstandard async filesystem where enumeration is very expensive and irrelevant matches are common.

It's easier to implement this by having access to the PatternToken and not needing to reimplement AnyWithin/AnyExcept parsing.

Start by locating the PatternToken definition and the Pattern parsing entry points, then read how token visibility currently limits inspection. Make PatternToken publicly accessible so callers can inspect parsed pattern elements without reimplementing parsing, and verify the examples in the issue can use the public API.