Pre-compile macro output as `include!`

@rzvxa and I have been discussing various ways to make macros cheaper on compile time, for example the #[ast] macro.

I had another idea in this vein. How about this:

• For each type marked #[ast], ast_tools would output a separate file.
• If the type is in oxc_ast/src/ast/js.rs, then the codegen would write to oxc_ast/.macro_expansions/src/ast/js.rs/<line number>.rs.
• Content of those files would be the complete output after processing (i.e. alter the type def in same way #[ast] macro does now).
• #[ast] macro would expand to:

include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));

Example

Original type def:

// oxc_ast/src/ast/js.rs line 123
#[ast]
enum Foo<'a> {
    Bar(Box<'a, Bar>),
    Qux(Box<'a, Qux>),
    @inherits Donkey,
}

ast_tools writes to file:

// oxc_ast/.macro_expansions/src/ast/js.rs/123.rs
#[repr(C, u8)]
enum Foo<'a> {
    Bar(Box<'a, Bar>) = 0,
    Qux(Box<'a, Qux>) = 1,
    BigDonkey(Box<'a, BigDonkey>) = 2,
    SmallDonkey(Box<'a, SmallDonkey>) = 3,
}

#[ast] macro's implementation is very simple indeed:

#[proc_macro_attribute]
pub fn ast(_: TokenStream, _: TokenStream) -> TokenStream {
    // No `syn` here!
    r#"include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));"#.parse().unwrap()
}

Spans

I've not tested this out, so I don't know for sure it'll work. In particular, I wonder if spans/hygiene will get mashed up so Rust Analyser may lose "jump to definition" etc.

Maybe we can work around that by outputting a macro_rules! macro. e.g. ast_tools writes this to the file:

// oxc_ast/.macro_expansions/src/ast/js.rs/123.rs
macro_rules! macro_123 {
    (
        #[ast]
        enum $Foo:ty $life:lifetime {
            $Bar:ident($Box1:ident<'a, $Bar:ident>),
            $Qux:ident($Box2:ident<'a, $Qux:ident>),
            @$inherits:ident $Donkey:ident,
        }
    ) => {
        #[repr(C, u8)]
        enum $Foo $life {
            $Bar($Box1<'a, $Bar>) = 0,
            $Qux($Box2<'a, $Qux>) = 1,
            macro_123!(@$inherits $Donkey,)
        }
    }

    (@inherits Donkey,) => {
        BigDonkey(Box<'a, BigDonkey>) = 2,
        SmallDonkey(Box<'a, SmallDonkey>) = 3,
    }
}

Then #[ast] macro would output a token stream:

include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));
macro_123! {
<original input TokenStream>
};

That's a bit more involved, but still can be done just by concatenating TokenStreams, which is much cheaper than syn.

Actually I'm not sure how to get proc macro to output macro_123!, as the proc macro doesn't know the line number. Maybe proc macro has to hash the input stream and use that as the UID instead of line number. Hashing with FxHash is fairly cheap.

Or maybe it searches through the input TokenStream for the token struct or enum, and then the UID is the token that follows (which will be the type name).

What do you think, @rzvxa?

Start by examining the existing #[ast] implementation and the ast_tools code generation around oxc_ast/src/ast/js.rs. Investigate whether include! output preserves spans and Rust Analyzer navigation, and compare the direct expansion and macro_rules! approaches described here. Done means selecting and implementing a viable pre-compilation design with confirmed hygiene behavior.