markedjs / markedjs/marked

Support custom extensions "interrupting" built-in tokens

Open
#3,435 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

proposal
Dominant language
JavaScript
Stars
37.2k
Forks
3.7k
Avg merge
4d 3h
Merged PRs (30d)
19

Description

What pain point are you perceiving?.
Not sure the best way to describe this. So currently, custom extensions have the start property which we use to interrupt the paragraph element. But there are other tokens that are interruptable according to the Commonmark/GFM spec. For example, GFM Tables must end when they encounter another block-level token.

The difficulty comes with enforcing that rule for custom extensions. Say I make a new block-level token via custom extensions

{{block
....
}}

If this were placed immediately after a Table, the table would just consume it, because it does not interact with the start property in the same way that paragraph does. You could roll your own Table tokenizer that does nothing but except add a few more characters to the Rules regex, but this seems like a lot of effort just to make your extension compatible with GFM rules.

Describe the solution you'd like
I really don't know how this would be implemented, but the desire would be a way for an extension to signal which tokens it can interrupt. Or, maybe better the other way around, allow a token to specify which types of other tokens can interrupt it.

One thing to consider, is that each token is also a little different in terms of at what points it can be interrupted. Blockquotes can only be interrupted during the "lazy continuation" step. Paragraphs can be interrupted any time. Tables can only interrupted if the line starts without |. Not every token can be interrupted by the same kinds of tokens.

I kind of hacked my way around this for Tables using my own extension Marked-Extended-Tables by allowing the user to input "termination" regex that would be appended to the tokenizer and cause table to stop lexing on that line.

https://github.com/calculuschild/marked-extended-tables/blob/9e56b24598e07de71e225d6c50a50d40c366965f/src/index.js#L23-L25

Not sure if this is the easiest way to go about it, but the trickiest part is somehow applying that to the built-in tokens without just ending up rewriting every tokenizer anyway.

Mostly I'm just kind of stumped on any better way to do this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the extension start handling and how built-in block tokens decide when to stop consuming input. Compare the linked Marked-Extended-Tables termination-regex workaround with the CommonMark/GFM interruption cases described here. Done means defining and documenting an extension mechanism that handles token-specific interruption rules without rewriting every built-in tokenizer.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.