Select Extension priority
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 37.2k
- Forks
- 3.7k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 19
Description
Describe the feature
Custom extensions always take priority over default tokens. A way to specify where in the parsing sequence an extension should run so certain default tokens don't get overridden.
Why is this feature necessary?
Sometimes there are ambiguous cases in Markdown where the result depends on which token gets parsed first. Example, tables without a starting pipe:
<div> header
|:---:|
cell
This is not be parsed as a table, because HTML is parsed first. However if someone writes a custom table extension for example, suddenly tables take priority and this is rendered as a table. The user would want the extension to execute before the default table tokenizer but after the HTML tokenizer.
Describe alternatives you've considered
It is possible to work around this by, in the extension tokenizer, pre-parsing the string and checking if any higher-priority tokens appear. However this adds extra steps and results in the same line potentially being parsed multiple times.
I'm picturing something like breaking the lexer.blockTokens() and lexer.inlineTokens() functions into an array of tokenizers and calling each one in sequence, and a user making an extension would somehow be able to inject their extension into this array (added to the front by default), though I imagine calling each tokenizer from an array instead would cause some slowdown.
In fact I think that was the original approach to the Extensions feature when I was making it, some kind of list of tokenizers, but the slowdown was too much. But now that it's been out for a while I am starting to run up against this limitation more and more.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the lexer.blockTokens() and lexer.inlineTokens() entry points, then inspect the Extensions feature and its tokenizer ordering. Define how custom tokenizers should be positioned relative to default tokenizers, and consider the stated parsing-speed concern. Done means extensions can select their priority so the example ordering is preserved without requiring repeated parsing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, markdown
- Domain
- compilers
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100