JetBrains / JetBrains/Grammar-Kit

Should we have comments in BNF?

Open
#337 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
771
Forks
137
Avg merge
11h 14m
Merged PRs (30d)
1

Description

PsiBuilder ignores whitespace and comment tokens but we have to use comment token in BNF just to have corresponding IElementType generated.

Even the official documentation for [Custom Language](https://plugins.jetbrains.com/docs/intellij/grammar-and-parser.html#define-the-grammar) has it as example: `private item_ ::= (property|COMMENT|CRLF)`.

In my own grammar I've ended up with an 'orphan' rule and it works fine: `private comment ::= COMMENT`

If comment tokens doesn't add anything to a parser, shouldn't we stop using them as we do for whitespace tokens. Of course they are necessary for corresponding IElementType generation but we could use an 'orphan' rules for them or even have a predefined `com.intellij.psi.TokenType`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the Grammar-Kit handling of BNF comment tokens and the IntelliJ grammar-and-parser documentation linked in the issue. Compare the documented `COMMENT` usage with the proposed orphan rule and predefined `TokenType` alternatives; done means an agreed direction for comment-token handling is documented or implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.