Rust-GCC / Rust-GCC/gccrs

`DelimTokenTree`-based parsing

Open
#3,791 3 comments 0 reactions 1 assignee View on GitHub

@powerboat9 is already working on this.

Since May 10, 2025.

Dominant language
C++
Stars
2.9k
Forks
231
Avg merge
20h 2m
Merged PRs (30d)
66

Description

For a future project, it would be nice to convert the parser to run on DelimTokenTree, rather than a stream of tokens. It would make parsing simpler and help catch edge cases like #3790.

Implementation wise, we'd probably want to do a first pass through the list of tokens, verifying that there aren't any mismatched delimiters. During that pass we could also cache distances between delimiter token pairs, sorted by left delimiter location. For example, [ ( { a } b ) c d ] could produce the distance cache [8, 4, 1]. Then we could have something like a ManagedTokenSource, say, TreeSource, that gives out either tokens or delimited token trees (via new instances of TreeSource). It would only have to store an iterator into the list of tokens, an ending iterator into the list of tokens, and an iterator into the distance cache.

Producing an initial TreeSource would be simple. Additionally, spinning off TreeSource instances for nested delimited pairs would be cheap, and the distance cache would make advancing the original TreeSource instance past the nested delimited pair cheap as well.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.