JetBrains / JetBrains/Grammar-Kit

Idea: TokenSet instead of vararg array of token elements

Open
#296 0 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Java
Stars
771
Forks
137
Avg merge
11h 14m
Merged PRs (30d)
1

Description

The generated parser uses varargs to find a matching token. For heavily used rules this is allocating a lot of memory.
I'm not sure if this is feasible, but creating a static TokenSet for the tokens instead of allocating an array for each invocation could reduce allocations (and thus GC) a lot.
Thank you for your consideration!

```java
public static boolean process_substitution(PsiBuilder b, int l) {
if (!recursion_guard_(b, l, "process_substitution")) return false;
// here, a static TokenSet could be passed to avoid the implicit array allocation
if (!nextTokenIsFast(b, INPUT_PROCESS_SUBSTITUTION, OUTPUT_PROCESS_SUBSTITUTION)) return false;
//...
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the generated parser code and the nextTokenIsFast entry point shown in the example, then trace how vararg token elements are passed for heavily used rules. Measure allocation behavior before and after the proposed approach; done means matching-token checks avoid per-invocation array allocation without changing parser behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
compilers, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.