Inability to specify EOF in TokenParsers.Parser

As discussed in https://github.com/scala/scala-parser-combinators/issues/397 by @martingd, scanners use their atEnd predicate instead of reporting EofCh or the built-in Tokens.EOF:

Note about EofCh

First of all, it seems scala.util.parsing.combinator.lexical.Scanners.Scanner will never read past the end of the its in: Reader[Char] because it stops at in.atEnd so the token and whitespace parsers in scala.util.parsing.combinator.lexical.StdLexical will never encounter the EofCh character. However, we have left the mentions of EofCh in the token and whitespace parsers to avoid breaking something we might not fully understand.

The fixes implemented for strings and such work for those cases, but it seems like it's impossible for a combinator defined in a TokenParsers.Parser object to determine if its scanner has reached the end-of-file without having access to the Scanner instance; in particular, any combinator that asserts the presence of EOF will fail because the token can never emit.

It is, of course, possible that I'm using the library in a way different from how it was intended to be used. If it would be helpful, I can modify and publish my code (I'm working on a reference solution, which I don't want to post so my students can't plagiarize it). On that note, would you mind if I worked on some examples that could help with documentation? How stable is the API (https://github.com/scala/scala-parser-combinators/issues/236, for example, suggested that there might be another way to represent data types)?

Start by reading TokenParsers.Parser, Scanners.Scanner, and StdLexical, together with the discussion in issue 397, to understand how atEnd, EofCh, and Tokens.EOF interact. The issue is resolved when a supported, documented way to assert EOF from a token parser is defined and its behavior is demonstrated or tested.