Specify and implement lexer tracing for debugging support

This is a broad feature for discussing the type of trace information logged by the lexer for debugging purposes as well as the log format. The trace information should support (at minimum) a range of to-be-specified debugging features. I'll work on opening a wiki article for planning this feature in further detail, and use this issue for tracking progress.

Start with the issue's planned wiki article and use it to define the lexer trace information and log format before implementation. The work is done when the debugging requirements are specified and the lexer tracing feature is implemented against that specification.

Written by the indexing model from the issue text.