Pre-RFC: Making macros and consteval dynamic, composable, and recursive

Rust has two important features for compile-time evaluation: macros and consteval functions. The way these two features interact is not bidirectionally composable, which leads to some limitations in the evaluation of macros and consteval functions.

Macro compilation happens in a single pass; that is, all macros in the source code are processed recursively until they are all expanded. Consteval functions run after macro expansion because consteval functions need to be converted to MIR in order to be interpreted. In other words, we currently don't macro expand one part of the AST separate from another part of the same AST. This means you can call macros from consteval functions, but not the other way around. Note that this is different from putting the consteval function into the macro output, because it gets run at a different time. It's also different from calling consteval functions from a procedural macro, because we are instead directly calling the consteval function dynamically at macro evaluation time, like a non-consteval function.

I want to make macros and consteval functions more powerful by eliminating this restriction, so that consteval functions can be called directly from the implementation of a macro. This would look something like a simplified version of MacoCaml's composable/compilable macros. As we currently interpret all consteval code as MIR, this would require changes at the HLIR level and above. In the context of this issue, "compilable macros" means macros which are compilable to MIR. Logically, it shouldn't matter whether the MIR is interpreted, compiled, or some mix of the two.

This will require syntax changes at some point to take advantage of the feature, but for now, I wanted to implement something simple and internal to the compiler to test if the core idea works in the Rust compiler.

The kinds of features this would unlock could be very useful. Here are some ideas:

• Passing a function pointer to a consteval function to a macro
  • The macro can call the consteval function with a function pointer that drives the generation of the code in the macro. This passes control of the macro evaluation to the consteval function.
• Splitting up evaluation of different macros in the same file
  • For example, we could evaluate the macros inside consteval functions, then pass that consteval function into macros outside of the consteval functions. We even do this in topological order.
  • This would help execute consteval/macros in parallel, even for large files, because it splits them up into workable chunks.
  • This would help compile hot consteval functions and macros, while interpreting others.
• This could help make crates like https://crates.io/crates/eval-macro more powerful

I have many ideas built on top of this feature, mainly targeted at generating types and functions in consteval functions and macros at compile time. I see this as the foundation to making significantly more powerful macros and consteval functions.

Please leave any early feedback you have about the idea. Thank you!

Start with the issue's description of macro expansion, consteval functions, HLIR, and MIR, then read the linked MacroCaml paper for context. The initial milestone is an internal compiler experiment that tests whether macros can directly call consteval functions; syntax changes are explicitly deferred.