[v2] Repo + package architecture

I've been thinking a lot about our architecture, how we split themes, plugins, languages etc. I think I agree with @DmitrySharabin's suggestion of having core plugins that live in the core repo, and it seems to be the direction many other plugin-heavy projects do it.

It seems that something like this might work well:

• Monorepo for core + languages + core plugins + core theme (just the one)
• Separate repos for non-core plugins and other themes
• Separate repo for website, which pulls in plugin readmes through build script

The alternative that many others follow is a single monorepo for all packages (with no distinction of official and not) + a separate repo for the website.

Going through our plugins, I'd suggest privileging these as official:

Plugin	Core?	Note
autolinker/	❌
autoloader/	✅	Perhaps this should not be a plugin at all
command-line/	❌
copy-to-clipboard/	✅
custom-class/	✅
data-uri-highlight/	✅	Perhaps this should even be enabled by default
diff-highlight/	❌	Ideally, this should be a language definition, see below
download-button/	❌
file-highlight/	❌
filter-highlight-all/	✅	Perhaps this should not be a plugin at all, but just a hook in the core? Though I haven't gone through it extensively.
highlight-keywords/	❌
inline-color/	❌
jsonp-highlight/	❌
keep-markup/	✅
line-highlight/	❌
line-numbers/	✅
match-braces/	❌
normalize-whitespace/	✅
previewers/	❌
show-invisibles/	✅
show-language/	✅
toolbar/	❌
treeview-icons/	❌
unescaped-markup/	✅
wpd/	🗑️	See below

Notes for right now

Remove WPD plugin

WPD is no longer a thing. A general plugin for linking tokens to docs (Doc Links?), including default mappings to MDN would be helpful though!

It could be extensible so that other plugins can add mappings for other docs sites.

Autoloading should become the default

Autoloading is incredibly common, and if done well, it should not get in the way: use what languages you got, autoload any non-optional languages you don't have. Also, it should work for both the browser and Node, instead of having a separate Node loader.

Making it part of the core will simplify many things, as we can now lean on it for other tasks. E.g. highlighting languages within other languages can just key on language-xxx language tokens and autoload them.

Instead of having to opt-in, users can opt-out of autoloading, by setting a config option.

Notes for later

These might need to wait until v3, but recording them now for posterity.

data-uri-highlight and diff-highlight are really languages, not plugins

The fact that they need to be defined as plugins highlights that the language API is not flexible enough to accommodate use cases.

Data URIs should probably be highlighted by default, with the following architecture:

• Instead of hardcoding the mapping of MIME types to languages, each language definition would define the MIME types that should be mapped to it, just like it defines its aliases.
• URI becomes a language definition, that CSS and HTML can use
• When a data URI, the contents are highlighted according to the matched language.

diff-highlight is complicated by the fact that there are two use cases here:

• Just diff highlighting, with no additional language highlighting. This is already doable easily with a diff language definition.
• Diff highlighting + highlighting of the actual language. I'm not sure how this works on a high level as I haven't gone through the code. Does it highlight the code within each insertion and deletion separately?

One feature that would facilitate both of these is dynamic tokens. Instead of language definitions having to hardcode which parts of the code are highlighted using another language (e.g. the markup language highlights code within <style> tags as CSS), be able to make that decision dynamically.

Start by reviewing the current repository layout and the plugin directories named in the issue, including autoloader, data-uri-highlight, diff-highlight, and WPD. Compare the proposed monorepo and split-repository options with the existing package and website arrangements. Done would require an agreed architecture and an explicit migration scope, rather than only a list of preferences.