macro future proofing: extend analysis to detect unfortunate interaction across arms

spawned off of https://github.com/rust-lang/rust/issues/30531#issuecomment-170080470

(transcribed relevant material from that comment below.)

when we released 1.0, we set some parameters for what sorts of changes we could make to the Rust grammar. In particular, for things like expressions and types, which can be referenced from macro definitions, we established (in RFC 550) a set of legal "follow" tokens that must never be added to the grammar. For example, the follow set for an expression was defined as =>, ,, and ;. These sets were derived by looking at the existing Rust grammar and finding expression terminators we were already committed to: for example, expressions can already be followed by , in a tuple expression or function call. Under those rules, the change to implement type ascription is legal.

RFC 550 also defined a static analysis that was intended to reject macro definitions which might be affected by (legal) changes to the expression grammar. For example, that analysis would reject a macro arm such as $e:expr : $t:ty because, there, $e is followed by :, which is not in the follow set for an expression. This means that if we were to add something like type ascription, the macro would break, and we did not want that to happen. However, as this issue demonstrates, that static anlysis contains a "soundness bug" when it comes to multiple macro arms.

In the medium to long term, we need to patch RFC 550 to correctly account for multiple macro arms. In the short term, however, macro authors will have to look at their own macros to check whether they might be broken by changes to the extension grammar.

Start with RFC 550, especially its follow-set rules and static analysis, then read the linked rust-lang/rust issue comment for the multiple-arm soundness case. The work is done when the analysis accounts for interactions across macro arms and rejects definitions that could break under legal grammar changes.