markedjs / markedjs/marked

Select Extension priority

Open
#3,590 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

proposal
Dominant language
JavaScript
Stars
37.2k
Forks
3.7k
Avg merge
4d 3h
Merged PRs (30d)
19

Description

Describe the feature
Custom extensions always take priority over default tokens. A way to specify where in the parsing sequence an extension should run so certain default tokens don't get overridden.

Why is this feature necessary?
Sometimes there are ambiguous cases in Markdown where the result depends on which token gets parsed first. Example, tables without a starting pipe:

<div> header
|:---:|
cell

This is not be parsed as a table, because HTML is parsed first. However if someone writes a custom table extension for example, suddenly tables take priority and this is rendered as a table. The user would want the extension to execute before the default table tokenizer but after the HTML tokenizer.

Describe alternatives you've considered
It is possible to work around this by, in the extension tokenizer, pre-parsing the string and checking if any higher-priority tokens appear. However this adds extra steps and results in the same line potentially being parsed multiple times.

I'm picturing something like breaking the lexer.blockTokens() and lexer.inlineTokens() functions into an array of tokenizers and calling each one in sequence, and a user making an extension would somehow be able to inject their extension into this array (added to the front by default), though I imagine calling each tokenizer from an array instead would cause some slowdown.

In fact I think that was the original approach to the Extensions feature when I was making it, some kind of list of tokenizers, but the slowdown was too much. But now that it's been out for a while I am starting to run up against this limitation more and more.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the lexer.blockTokens() and lexer.inlineTokens() entry points, then inspect the Extensions feature and its tokenizer ordering. Define how custom tokenizers should be positioned relative to default tokenizers, and consider the stated parsing-speed concern. Done means extensions can select their priority so the example ordering is preserved without requiring repeated parsing.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, markdown
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.