pest-parser / pest-parser/pest

[feature] Generating constant str rules

Open
#718 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

derive enhancement generator
Dominant language
Rust
Stars
5.4k
Forks
307
Avg merge
8h 23m
Merged PRs (30d)
5

Description

Problem

Often the grammar requires some constant strings in it (keywords, operators, etc). These can be introduced as a separate rule, example:

return_type_operator = { "->" }
function_keyword = { "fn" }

and then reused in some other rules. When parsing and then generating the AST, these constant tokens are lost (which is obviously a good thing). But when we want to serialize the AST back to its code form we need these constant tokens. Since pest does not generate these constant strings as public (they are inlined), one could not reuse them. The final solution is to redefine them, for example by attaching them to the AST nodes:

impl FunctionNode {
	const KEYWORD: &str = "fn";
}

This leads to a problem when we want to change a constant string; we need to remember to change it in both places.

Proposal

Pest could detect const rules and generate constants for them, for example:

// After #[derive(Parser)]
impl Constants {
	const return_type_operator: &str = "->";
	const function_keyword: &str = "fn";
}

I see two paths that can be taken here:

  1. If a rule is composed of a single literal string, we generate a constant for it
  2. If the rule can be evaluated as const, we generate a constant for it

Path 2 is obviously preferred, but harder to implement: then there would be a need of some const eval engine. For instance, a rule is constant if all rules used inside are constant, if the repetition count is constant, etc. Then a the rule would be have to be evaluated to a single final constant string.

I don't know the codebase at all, but I'm pretty sure I would be able to implement path 1 and send a PR for it if it is wanted. I would only worry about whether such a primitive feature would feel incomplete compared to path 2.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file, test, or entry point is named in the issue. Begin by locating the derive-parser code generation path and how single-literal rules are represented; completion should generate reusable constants for those rules without duplicating their literals, with tests covering the proposed API.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.