MatchingEngine Capabilities and Roadmap

Details

TODO

Regex feature status

The matching engine supports (modulo bugs) the following

• Basic constructs: concatenation, alternation, non-capturing grouping, etc.
• Literal constructs: scalar literals, quotes,
• Character classes, custom and built-in
  • Including ranges, nested custom character classes, some set operations (e.g. subtraction), inversion, etc
• Quantification
  • eager, reluctant, possessive x* x*? x*+
  • bounded and unbounded x+ x{n,m} x{n,} x{,m} x{n} (and kind variants)
• Character properties: named characters, general category, most UCD properties
• Assertions: built-in and custom
  • Including anchors such as $, ^
  • Including custom look-ahead assertions
• Arbitrary consumer call-outs (Input, Range<Input.Index>) -> Input.Index?
  • This is the basic extension point for library-driven pattern matching
  • This is how character classes are currently implemented
• Arbitrary assertion call-outs (Input, Input.Index, Range<Input.Index>) -> Bool
  • This is how anchors are currently implemented
  • Note the provided bounds, as assertions often deal with boundary conditions
• Arbitrary value-producing callouts (CustomRegexComponent)
• A function call stack for recursive PEG-style grammars
  • Note: Backtracking properly manages this by restoring stack position
  • Note: This is only very very lightly tested (mostly for PEGs)
  • TODO: Hook up to (?R), etc.
• Captures
• Backreferences
• Scripts and some missing Unicode scalar properties

The following has some corner-case known bugs in it

• Backtracking to completely different function call stack
  • Currently we restore a stack index, but it's not clear if we need to restore entire stack

The following are currently unsupported

• All Unicode scalar properties
• Atomic grouping
  • Will need to figure out how best to play with the rest of the stdlib here
• Custom look-behind assertions
• Script runs and PCRE-style call outs
  • We have engine support, but isn't hooked up to syntax
  • We will likely want something strongly-typed and better checked
• Matching options
  • Things like case-insensitivity, semantic mode switching, etc
• Subpatterns
• Conditional patterns
  • (awaiting parser support)
• Oniguruma style absent functions
• Keep/reset (\K)

The following is undetermined

• Grapheme-semantic mode switching and behavior/design
• Options, especially controlling backtracking
• Provisioning for the interpreter
• How best to do word-boundary analysis

Performance

TODO