rust-lang / rust-lang/rust-analyzer
Reintroduce opaque IDs for span data in token trees
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 16.9k
- Forks
- 2.2k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 72
Description
The token-map rewrite changed our token trees from having opaque ids that map input to output tokens to containing spans which consist of a relative text span and a SyntaxContextID (hygiene). This more closely models how spans behave but are less incremental. The change did go into the right direction, but in hindsight it feels like we should instead keep the opaque IDs and produce an additional mapping from those ids to the actual span data, reasons being:
- This potentially allows us to skip re-expanding macros if all that changed was trivia tokens inside the input. These changes only change the spans which are irrelevant to the resulting token trees (except for proc-macros that read out the span info, but that can be tracked)!
- It potentially deduplicates data. If two tokens use the same span, the span is duplicated in memory where as a mapping could potentially allow for deduplication through a separate table.
- It allows us to attach additional data to tokens in side tables without wasting a bunch of memory if the data is only relevant for a few tokens.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the token-map rewrite and the token-tree implementation described in the issue, then trace how relative text spans and SyntaxContextID values flow through macro expansion. Compare the current representation with the former opaque-ID design and determine the required mapping and side-table boundaries; done means token trees use opaque IDs while preserving the needed span data and incremental behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100