rust-lang / rust-lang/rust-analyzer
Re-think/design our `TokenTree` model
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 16.9k
- Forks
- 2.2k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 72
Description
I think it might be a good idea to step back for a second, clarify the current status quo and check what we can improve more structurally
Right now we have the following setup:
- parsing normal source files was lex the source string into tokens, then parse that into our rowan AST.
- parsing macro expansions, we expand the macro which yields a
TokenTree, then that is being parsed into the AST via syntax bridge (which basically attempts to retokenize)- macro expansion, we turn the AST token tree of its input(s) into a
TokenTreevia syntax bridge
- what we do differs a bit dependong on the kind of macro
- for MBE macros we then do the matching and transcribing against this
TokenTree- for proc-macros we serialize this format into json, throw that at the proc-macro server, deserialize it again and transform that to the proc-macro types on demand (as required by the bridge)
- for builtin macros it depends, most stuff just works on the
TokenTreestuff directly as most builtins are simple, except for derives which re-parse theTokenTrees into the AST
- noticable eager macros parse the TokenTree back fully, then descends the tree to find any new macro call, expands that, transforms that back into the AST and patches that into its AST until the AST is fully expanded turning it back into a
TokenTree- that leaves us with a final
TokenTreeThat means we currently have two types of syntax trees, our AST (or well, CST) which is what the lowering and the IDE works with and the
TokenTreemodel which is what macro expansion work with.We should ideally lay out the requirements here to see if we actually need the
TokenTree(my gut tells me that I think we do, just that its shape is not ideal). I haven't given this much thought otherwise right now, and likely won't the coming week but to clarify I'd like to have a proper plan laid out with reasoning behind it before we do any major change here then as this is touching on a fundamental system in rust-analyzer.
Originally posted by @Veykril in #18764
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by mapping the AST/CST and TokenTree paths described in the issue, including syntax bridge, macro expansion, MBE matching, proc-macro serialization, and builtin derives. Document the requirements and trade-offs for retaining or reshaping TokenTree, then produce a reasoned plan before any implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100