rust-lang / rust-lang/rust-analyzer

Reintroduce opaque IDs for span data in token trees

Open
#17,323 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-macro A-perf C-enhancement
Dominant language
Rust
Stars
16.9k
Forks
2.2k
Avg merge
1d 12h
Merged PRs (30d)
72

Description

The token-map rewrite changed our token trees from having opaque ids that map input to output tokens to containing spans which consist of a relative text span and a SyntaxContextID (hygiene). This more closely models how spans behave but are less incremental. The change did go into the right direction, but in hindsight it feels like we should instead keep the opaque IDs and produce an additional mapping from those ids to the actual span data, reasons being:

  • This potentially allows us to skip re-expanding macros if all that changed was trivia tokens inside the input. These changes only change the spans which are irrelevant to the resulting token trees (except for proc-macros that read out the span info, but that can be tracked)!
  • It potentially deduplicates data. If two tokens use the same span, the span is duplicated in memory where as a mapping could potentially allow for deduplication through a separate table.
  • It allows us to attach additional data to tokens in side tables without wasting a bunch of memory if the data is only relevant for a few tokens.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the token-map rewrite and the token-tree implementation described in the issue, then trace how relative text spans and SyntaxContextID values flow through macro expansion. Compare the current representation with the former opaque-ID design and determine the required mapping and side-table boundaries; done means token trees use opaque IDs while preserving the needed span data and incremental behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.