PrismJS / PrismJS/prism

Use the token overlay helper for the remaining structure-merging code

Open
#4,111 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
13k
Forks
1.4k
Avg merge
15h 36m
Merged PRs (30d)
3

Description

"Merge one tokenized structure with another over the same text" is reimplemented or worked around in many places. The $inner selectors PR adds the primitives (splitTokenStream() and insertTokens() in src/util/token-stream.js) and uses them for meta-languages. These are the remaining instances found by scanning the repo, roughly in order of payoff:

DOM / string level today, should be token level

  • keep-markup (src/plugins/keep-markup/keep-markup.js:62-168): records text offsets of existing elements, then re-wraps them after highlighting with Range.extractContents(), which cuts through token spans. → wrap ranges on after-tokenize with splitTokenStream. Its node-identity test (tests/plugins/keep-markup/test.js) must keep passing. normalize-whitespace's special branch for keep-markup (normalize-whitespace.js:213-223) goes away with it.
  • command-line (src/plugins/command-line/command-line.js:60-131): stashes output lines and strips continuation prefixes on env.code, re-inserts on the highlighted HTML string split by \n (breaks any multi-line token). → the same shape as diff: select the command lines.
  • markdown code blocks after autoload (src/languages/markdown.js:298-337): re-highlights via element.innerHTML = Prism.highlight(element.textContent).
  • inline-color (src/plugins/inline-color/inline-color.js:92): recovers text by stripping tags from HTML.

Per-line / per-node grammar workarounds

  • xml-doc /// lines (src/languages/xml-doc.js:11-26), javadoclike / jsdoc * prefixes (javadoclike.js:5-15, jsdoc.js, javascript.js:93-98): every pattern is /m and anchored per line because the prefixes are never removed, so nothing can match across lines. → select the prefix-stripped lines, tokenize as one document (exactly what diff does now).
  • markdown blockquotes (markdown.js:49-52): > prefix per line, the remainder is not re-parsed as markdown.
  • shell-session (src/languages/shell-session.js:58-71): each command highlighted as bash separately; output is one opaque token.

Per-string overlays that miss matches across token boundaries

  • tokenizeStrings() (src/shared/tokenize-strings.js) and its users show-invisibles, autolinker, data-uri-highlight: a URL or data URI split by a token boundary is never matched. → :text-style selection of all strings at any depth, highlighted as one whole.

Same algorithm twice

  • jsx and xquery walkTokens (src/languages/jsx.js:24-109, src/languages/xquery.js:8-105): concatenate selected nodes into plain-text tokens. → select + an empty inner language.

Ordering hazards that exist only because these overlays don't compose: command-line.js:82-84, line-highlight.js:348-370.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/util/token-stream.js and compare the existing diff and meta-language uses of splitTokenStream() and insertTokens(). Work through the listed keep-markup, command-line, language, and overlay call sites, preserving the keep-markup node-identity test and checking the noted ordering hazards. Done means the remaining structure-merging work uses token-level overlays rather than the listed DOM, HTML-string, or per-line workarounds.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.