JetBrains / JetBrains/Grammar-Kit
Idea: TokenSet instead of vararg array of token elements
- Dominant language
- Java
- Stars
- 771
- Forks
- 137
- Avg merge
- 11h 14m
- Merged PRs (30d)
- 1
Description
The generated parser uses varargs to find a matching token. For heavily used rules this is allocating a lot of memory.
I'm not sure if this is feasible, but creating a static TokenSet for the tokens instead of allocating an array for each invocation could reduce allocations (and thus GC) a lot.
Thank you for your consideration!
```java
public static boolean process_substitution(PsiBuilder b, int l) {
if (!recursion_guard_(b, l, "process_substitution")) return false;
// here, a static TokenSet could be passed to avoid the implicit array allocation
if (!nextTokenIsFast(b, INPUT_PROCESS_SUBSTITUTION, OUTPUT_PROCESS_SUBSTITUTION)) return false;
//...
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the generated parser code and the nextTokenIsFast entry point shown in the example, then trace how vararg token elements are passed for heavily used rules. Measure allocation behavior before and after the proposed approach; done means matching-token checks avoid per-invocation array allocation without changing parser behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- compilers, devtools
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100