rust-lang / rust-lang/regex

regex-syntax: some way to retain the AST Span of some punctuation marks?

Open
#1,271 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
4k
Forks
534
PR merge metrics
No merged PRs in 30d

Description

Consider:

use regex_syntax::ast::parse::ParserBuilder;

fn main() {
    let parse = |pattern| {
        ParserBuilder::new()
            .ignore_whitespace(true)
            .build()
            .parse_with_comments(pattern)
            .unwrap()
    };

    let wc_1 = parse("a #c\n|b");
    let wc_2 = parse("a|#c\n b");
    assert_ne!(wc_1, wc_2);
}

The comment #c is attached to different alternatives in the two regex, but the parse output of both are equivalent:

WithComments { 
    ast: Alternation(Alternation { 
        span: Span(Position(o: 0, l: 1, c: 1), Position(o: 7, l: 2, c: 3)), 
        asts: [
            Literal(Literal { 
                span: Span(Position(o: 0, l: 1, c: 1), Position(o: 1, l: 1, c: 2)), 
                kind: Verbatim, 
                c: 'a' 
            }), 
            Literal(Literal { 
                span: Span(Position(o: 6, l: 2, c: 2), Position(o: 7, l: 2, c: 3)), 
                kind: Verbatim, 
                c: 'b' 
            })
        ] 
    }), 
    comments: [
        Comment { 
            span: Span(Position(o: 2, l: 1, c: 3), Position(o: 5, l: 2, c: 1)), 
            comment: "c" 
        }
    ] 
}

$$\overbrace{\overbrace{\Huge\color{red} \texttt{a}\mathstrut}^{\textrm{Literal(0..1)}}{\Huge\color{blue}\texttt{␣ }}\underbrace{\Huge\color{green}\texttt{\# c ↵}\mathstrut}_{\textrm{Comment(2..5)}}{\Huge\color{blue}\texttt{ |}}\overbrace{\Huge\color{red}\texttt{b}\mathstrut}^{\textrm{Literal(6..7)}}}^{\textrm{Alternation(0..7)}}$$

Without knowing the span of the | punctuation we cannot know if the comment should belong to a or b from parse_with_comments() alone. We have to refer back to the original pattern. At which point perhaps it is easier to just write the parser ourselves 🤷

I think the Ast type itself should include the Span of these marks when their position cannot be inferred, like the | in a|b|c or the , in a{3,100}.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with regex_syntax::ast::parse::ParserBuilder and parse_with_comments, reproducing the two patterns shown in the issue. Read the Ast and span representations to determine how punctuation such as | and , could be retained. Done means the parse output preserves enough punctuation positions to distinguish which alternative owns a comment.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.