rust-lang / rust-lang/rust-analyzer

Re-think/design our `TokenTree` model

Open
#18,808 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-macro A-proc-macro C-Architecture
Dominant language
Rust
Stars
16.9k
Forks
2.2k
Avg merge
1d 12h
Merged PRs (30d)
72

Description

I think it might be a good idea to step back for a second, clarify the current status quo and check what we can improve more structurally

Right now we have the following setup:

  • parsing normal source files was lex the source string into tokens, then parse that into our rowan AST.
  • parsing macro expansions, we expand the macro which yields a TokenTree, then that is being parsed into the AST via syntax bridge (which basically attempts to retokenize)
  • macro expansion, we turn the AST token tree of its input(s) into a TokenTree via syntax bridge
    • what we do differs a bit dependong on the kind of macro
      • for MBE macros we then do the matching and transcribing against this TokenTree
      • for proc-macros we serialize this format into json, throw that at the proc-macro server, deserialize it again and transform that to the proc-macro types on demand (as required by the bridge)
      • for builtin macros it depends, most stuff just works on the TokenTree stuff directly as most builtins are simple, except for derives which re-parse the TokenTrees into the AST
        • noticable eager macros parse the TokenTree back fully, then descends the tree to find any new macro call, expands that, transforms that back into the AST and patches that into its AST until the AST is fully expanded turning it back into a TokenTree
    • that leaves us with a final TokenTree

That means we currently have two types of syntax trees, our AST (or well, CST) which is what the lowering and the IDE works with and the TokenTree model which is what macro expansion work with.

We should ideally lay out the requirements here to see if we actually need the TokenTree (my gut tells me that I think we do, just that its shape is not ideal). I haven't given this much thought otherwise right now, and likely won't the coming week but to clarify I'd like to have a proper plan laid out with reasoning behind it before we do any major change here then as this is touching on a fundamental system in rust-analyzer.

Originally posted by @Veykril in #18764

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by mapping the AST/CST and TokenTree paths described in the issue, including syntax bridge, macro expansion, MBE matching, proc-macro serialization, and builtin derives. Document the requirements and trade-offs for retaining or reshaping TokenTree, then produce a reasoned plan before any implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.