A plugin for splitting tokens into lines

Motivation
Recently, I wanted to try using Prism in a content-editable based editor. I quickly realized that Prism’s APIs on their own are a poor fit, partly because it does not split lines into multiple elements, preferring to create a DOM structure composed exclusively of inline elements and whitespace. This makes things difficult because Firefox has lots of issues working with newline-delimited editable DOM, especially when selecting parts of the document. Additionally, having all elements be inline can cause strange, inefficient mutations, which makes normalizing content-editable changes much more costly.

Description
Ultimately, I decided the fastest way to get Prism working with content-editables would be to figure out a way to split the tokens returned from Prism.tokenize() into lines, and to use that output to render to the DOM on my own. I didn’t see any plugins or third-party libraries which did this for me, so I ended up writing the following code (TypeScript, sorry).

import Prism from 'prismjs';
import type {Token} from 'prismjs';

function wrapContent(
  content: Array<Token | string> | Token | string,
): Array<Token | string> {
  return Array.isArray(content) ? content : [content];
}

function unwrapContent(
  content: Array<Token | string>,
): Array<Token | string> | string {
  if (content.length === 0) {
    return '';
  } else if (content.length === 1 && typeof content[0] === 'string') {
    return content[0];
  }

  return content;
}

function splitLinesRec(
  tokens: Array<Token | string>,
): Array<Array<Token | string>> {
  let currentLine: Array<Token | string> = [];
  const lines: Array<Array<Token | string>> = [currentLine];
  for (let i = 0; i < tokens.length; i++) {
    const token = tokens[i];
    if (typeof token === 'string') {
      const split = token.split(/\r\n|\r|\n/);
      for (let j = 0; j < split.length; j++) {
        if (j > 0) {
          lines.push((currentLine = []));
        }

        const token1 = split[j];
        if (token1) {
          currentLine.push(token1);
        }
      }
    } else {
      const split = splitLinesRec(wrapContent(token.content));
      if (split.length > 1) {
        for (let j = 0; j < split.length; j++) {
          if (j > 0) {
            lines.push((currentLine = []));
          }

          const line = split[j];
          if (line.length) {
            const token1 = new Prism.Token(
              token.type,
              unwrapContent(line),
              token.alias,
            );
            token1.length = line.reduce((l, t) => l + t.length, 0);
            currentLine.push(token1);
          }
        }
      } else {
        currentLine.push(token);
      }
    }
  }

  return lines;
}

export function splitLines(
  tokens: Array<Token | string>,
): Array<Array<Token | string>> {
  const lines = splitLinesRec(tokens);
  // Dealing with trailing newlines
  if (!lines[lines.length - 1].length) {
    lines.pop();
  }

  return lines;
}

The exported function splitLines() takes an array of tokens and strings, and returns an array of arrays of tokens and strings where each subarray represents a line. I’m not 100% on the invariants for the Token data structure; for instance, I’m not sure whether a token with a single token child should have its contents set to that child token or an array whose sole element is the child token. Nevertheless, it mostly works, and I’ve manually tested it against some code snippets which produce nested tokens in JavaScript/TypeScript.

I would be happy to contribute this code to Prism as a plugin, if the maintainers were interested. Alternatively, if anyone needs this code for themselves, feel free to take it, I release it under the same license as the Prism project.

Alternatives
Multiple people seem to be working on splitting Prism tokens by line. For instance, there is prism-react-renderer, which has a utility to split tokens (https://github.com/FormidableLabs/prism-react-renderer/blob/master/src/utils/normalizeTokens.js). I avoided this implementation because it defined its own Token interface. I thought it would be simpler to reuse the Prism Token constructor.

Additionally, there are these two pull requests (https://github.com/PrismJS/prism/pull/2389, https://github.com/PrismJS/prism/pull/2413). I avoided these implementations because they seem to work on the DOM after it has been rendered. I think having a way to split lines at the tokenization step would be much more useful going forward, though I know that Prism likes to do as much as it can without involving extra JavaScript.

Start with Prism.tokenize() and the Prism.Token data structure, then compare the proposed recursive splitLines() behavior with prism-react-renderer’s normalizeTokens.js and the implementations in pull requests 2389 and 2413. Done means providing a maintained plugin that splits nested token output into per-line arrays while preserving valid Prism token structure.