markedjs / markedjs/marked

let walkTokens replace the current token with multiple

Open
#3,844 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

proposal
Dominant language
JavaScript
Stars
37.2k
Forks
3.7k
Avg merge
4d 3h
Merged PRs (30d)
19

Description

Describe the feature
Currently, walkTokens lets you edit individual token properties as you walk the tree. However, you can't inject new tokens (except as children if the current token has a "tokens" property). I would like to take, say, an HTML token, and break it apart into multiple pieces as their own tokens. For example:

<div>content</div> is parsed into:

[
{type:"html", block:true, raw:"<div>\ncontent\n</div>", pre:false, text:"<div>\ncontent\n</div>"}
]

Which I would like to split into:

[
  {type:"html", block:true, raw:"<div>\n\n", pre:false, text:"<div>\n\n"}
  {type:"text", raw:"content", text:"content", escaped:false}
  {type:"html", block:true, raw:"</div>", pre:false, text:"</div>"}
]

Why is this feature necessary?
walkTokens is a great tool to tweak the token tree, particularly when you want to slightly change the behavior of a built-in token without reimplementing the entire tokenizer along with its regex. But it would be nice to have more control over the tree beyond just changing the existing tokens.

Describe alternatives you've considered
Fully re-implementing the tokenizer for built-in tokens does work, because you can break tokens down as much as you want, but extensions can quickly fall out-of-sync with Marked.js bug fixes, and requires a lot more maintenance work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing walkTokens traversal and the tokenizer behavior for HTML tokens. Determine how a current token could be replaced by multiple tokens while preserving tree traversal, then verify that the described HTML example produces the three-token result without breaking existing token edits.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.