Non-RegExp patterns

Motivation

Regexes are easy to use and powerful enough for most of the highlighting problems we face. However, sometimes regexes are not powerful enough.

Since some tokens can only be matched by a context-free grammar, we usually resorted to a number of tricks ranging from supporting a finite number of recursion steps to using the usually surrounding of the token. We use these tricks out of necessity but what we really need in those cases is some more powerful than regexes.

Description

I want to propose that we relax grammars to allow regex-like objects. The basic idea is that Prism's matching algorithm only uses the exec and lastIndex properties of RegExp objects, so any object that implements those properties can be used.

Speaking in types, I want to change:

interface GrammarToken {
  pattern: RegExp;
  ...
}
interface Grammar {
  [name: string]: RegExp | GrammarToken | Array<RegExp | GrammarToken>;
}

to:

interface RegExpLike { // RegExp trivially implements this interface
  lastIndex: number;
  exec(value: string): RegExpExecArray | null;
}
interface GrammarToken {
  pattern: RegExpLike;
  ...
}
interface Grammar {
  [name: string]: RegExpLike | GrammarToken | Array<RegExpLike | GrammarToken>;
}

This will allow us to implement custom matchers that can be more powerful than regexes.

Required changes to Prism Core

The only thing we have to change is how we try to enable the global flag here.

Edit: This idea require no changes to Core, if we require a global: true property in the RegExpLike interface.

Start by reviewing the grammar type definitions and the Prism Core matching behavior referenced near prism-core.js line 861. Determine how RegExpLike objects, including their exec, lastIndex, and global properties, should be represented and supported; done means grammars can use custom matchers without breaking existing RegExp behavior.