pest-parser / pest-parser/pest
[feature] Generating constant str rules
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 307
- Avg merge
- 8h 23m
- Merged PRs (30d)
- 5
Description
Problem
Often the grammar requires some constant strings in it (keywords, operators, etc). These can be introduced as a separate rule, example:
return_type_operator = { "->" }
function_keyword = { "fn" }
and then reused in some other rules. When parsing and then generating the AST, these constant tokens are lost (which is obviously a good thing). But when we want to serialize the AST back to its code form we need these constant tokens. Since pest does not generate these constant strings as public (they are inlined), one could not reuse them. The final solution is to redefine them, for example by attaching them to the AST nodes:
impl FunctionNode {
const KEYWORD: &str = "fn";
}
This leads to a problem when we want to change a constant string; we need to remember to change it in both places.
Proposal
Pest could detect const rules and generate constants for them, for example:
// After #[derive(Parser)]
impl Constants {
const return_type_operator: &str = "->";
const function_keyword: &str = "fn";
}
I see two paths that can be taken here:
- If a rule is composed of a single literal string, we generate a constant for it
- If the rule can be evaluated as const, we generate a constant for it
Path 2 is obviously preferred, but harder to implement: then there would be a need of some const eval engine. For instance, a rule is constant if all rules used inside are constant, if the repetition count is constant, etc. Then a the rule would be have to be evaluated to a single final constant string.
I don't know the codebase at all, but I'm pretty sure I would be able to implement path 1 and send a PR for it if it is wanted. I would only worry about whether such a primitive feature would feel incomplete compared to path 2.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named in the issue. Begin by locating the derive-parser code generation path and how single-literal rules are represented; completion should generate reusable constants for those rules without duplicating their literals, with tests covering the proposed API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100