Section Review: 31.1 Lexing and Parsing & 31.2 Macro Expansion

The following is a review of sections 31.1 Lexing and Parsing & 31.2 Macro Expansion.

What experience does the reviewer have with the material?

I've made changes to the parser before, and I've debugged issues and made some very small changes (that were never merged) inside of macro expansion. Overall, I'm somewhat familiar with parsing and quite unfamiliar with lexing and macro expansion.

Review

31.1

• "As of January 2021, the lexer and parser are undergoing refactoring to allow extracting them into libraries." - Is this still true?
• Macro expansion is mentioned but it's not defined yet. While it might feel obvious, it's nice to define the term.
• Missing some links to types like ParseSess which could also use a bit more explanation on what they are for.
• Span is mentioned but not defined
• The section mentions that macros are "set aside to be expanded". We should mention the term "placeholder" since that is used in the next section.
• There should probably some mention on error recovering for diagnostics. Much of the complexity of the parser is because of diagnostics. Here's a possible example.
• It's mentioned that lexers are often generated but that rustc's lexer is hand-written. Parsers can also be generated, but rustc's is not. This should be mentioned.
• We certainly don't want to write a text book on parsing, but it might be useful to define some common terms used in the codebase (e.g., "(non-)terminals")
• Perhaps the use of DUMMY_NODE_ID should be explained?

31.2

• What does the process of finding the placeholders to be expanded look like?
• What are "side-tables"? This gets mentioned without being defined.
• Doesn't cargo expand show source after expansion? Maybe we should mention that.
• The algorithm expansion mentions a queue, but rustc_expand has nothing named with queue. Is this InvocationCollector.invocations? Perhaps it would be nice to link this algorithm with the parts of the code that generally correspond to that part of the algorithm.
• set_expn_data is mentioned as a way to set data related to hygiene and then the section on hygiene is alluded to, but this is never really revisited.
• The section on other data structures doesn't feel very helpful at least at my point in trying to understand expansion. Not sure if this should be expanded, moved to the end of the section or something else.
• Three expansion hierarchies are mentioned. Perhaps the section that introduces the fact that there is 3 hierarchies could be cleaned up a bit. I was confused what the hierarchies were for and nothing indicated that they would be explained below.
• We should give a <!--date--> to the section on macros 2.0
• Overall this section is pretty dense, but the material itself is dense too. I have a better idea of how macro expansion works, but I still can't say I fully understand it (even at a high-level).

Start with the linked rustc-dev-guide sections 31.1 Lexing and Parsing and 31.2 Macro Expansion, then compare their references to rustc_parse/src/parser/stmt.rs, rustc_expand, InvocationCollector.invocations, and set_expn_data. Review each listed terminology, linking, algorithm, hierarchy, and date concern; done means the sections address the review points or clearly explain why they do not.