Use the token overlay helper for the remaining structure-merging code
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 13k
- Forks
- 1.4k
- Avg merge
- 15h 36m
- Merged PRs (30d)
- 3
Description
"Merge one tokenized structure with another over the same text" is reimplemented or worked around in many places. The $inner selectors PR adds the primitives (splitTokenStream() and insertTokens() in src/util/token-stream.js) and uses them for meta-languages. These are the remaining instances found by scanning the repo, roughly in order of payoff:
DOM / string level today, should be token level
- keep-markup (
src/plugins/keep-markup/keep-markup.js:62-168): records text offsets of existing elements, then re-wraps them after highlighting withRange.extractContents(), which cuts through token spans. → wrap ranges onafter-tokenizewithsplitTokenStream. Its node-identity test (tests/plugins/keep-markup/test.js) must keep passing. normalize-whitespace's special branch for keep-markup (normalize-whitespace.js:213-223) goes away with it. - command-line (
src/plugins/command-line/command-line.js:60-131): stashes output lines and strips continuation prefixes onenv.code, re-inserts on the highlighted HTML string split by\n(breaks any multi-line token). → the same shape asdiff: select the command lines. - markdown code blocks after autoload (
src/languages/markdown.js:298-337): re-highlights viaelement.innerHTML = Prism.highlight(element.textContent). - inline-color (
src/plugins/inline-color/inline-color.js:92): recovers text by stripping tags from HTML.
Per-line / per-node grammar workarounds
- xml-doc
///lines (src/languages/xml-doc.js:11-26), javadoclike / jsdoc*prefixes (javadoclike.js:5-15,jsdoc.js,javascript.js:93-98): every pattern is/mand anchored per line because the prefixes are never removed, so nothing can match across lines. → select the prefix-stripped lines, tokenize as one document (exactly whatdiffdoes now). - markdown blockquotes (
markdown.js:49-52):>prefix per line, the remainder is not re-parsed as markdown. - shell-session (
src/languages/shell-session.js:58-71): each command highlighted as bash separately; output is one opaque token.
Per-string overlays that miss matches across token boundaries
tokenizeStrings()(src/shared/tokenize-strings.js) and its users show-invisibles, autolinker, data-uri-highlight: a URL or data URI split by a token boundary is never matched. →:text-style selection of all strings at any depth, highlighted as one whole.
Same algorithm twice
- jsx and xquery
walkTokens(src/languages/jsx.js:24-109,src/languages/xquery.js:8-105): concatenate selected nodes intoplain-texttokens. → select + an empty inner language.
Ordering hazards that exist only because these overlays don't compose: command-line.js:82-84, line-highlight.js:348-370.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/util/token-stream.js and compare the existing diff and meta-language uses of splitTokenStream() and insertTokens(). Work through the listed keep-markup, command-line, language, and overlay call sites, preserving the keep-markup node-identity test and checking the noted ordering hazards. Done means the remaining structure-merging work uses token-level overlays rather than the listed DOM, HTML-string, or per-line workarounds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100