Discussion of Github Markdown Interoperability Level

_GitHub Flavored Markdown_ is specified in [github.github.com/gfm][1]. While this specification haven't changed much in recent years there appears to be two sources of interoperability issues:

* (i) Enhancement of existing GFM constructs.
* (ii) Changes in HTML sanitization rules.
* (iii) Undocumented GFM extensions.

The introduction of enhanced intepretations of existing GFM syntatical constructs (i) is fairly recent. We have not found any comprehensive specification of what constructs are enhanced.

The sanitization rules on GitHub used to be [`HTML::Pipeline`][2], however, around [2020][3] that seems to have changed. It's unclear if GitHub HTML sanitization rules are open source somewhere else, or simply undocumented. HTML sanitization is necessary because markdown allows for raw HTML. And when our HTML sanitization rules differs from GitHub users are likely to experince interoperability issues.

## Examples of (i) -- Mermaid support

GitHub have added native support for [mermaid diagrams][4] using code blocks, such that a block as follows:
```mermaid
...
```

Will be passed to mermaid and transformed into a picture that will be embedded instead of the code snippet.

This arguably a cool feature, currently not supported on pub.dev or any other Dart tooling. This is not covered by the [GFM specification][1], however, it can be reasonably argued that the markdown is still valid GFM, since new syntatic constructs have not been introduced.

## Examples of (i) -- Note/warning blockquotes (beta).

Another example of enhanced interpretation of existing syntactical blocks is [note/warning blockquotes][5]. Where a blockquote with a bold line is rendered as a note/warning block. As illustrated below:

> **Note**
> This is a note

## Examples of (ii) -- Changes in HTML sanitization rules

Use of the `` tag for specifying [images for both light and dark][6] themes is an example of a feature that doesn't degrade well when pub.dev uses the old HTML sanitization rules.

## Examples of (iii) -- Math in markdown on GitHub

GitHub recently launched [math syntax][7] on GitHub, using `$1+1=2$` for inline math and `$$1+1=2$$` for block math expressions. In both cases the language is LaTeX using MathJax. Unlike (i) and (ii), appears to be an undocumented extension of GFM.

Example: $\large\frac{2+2}{\sqrt{\pi}}$

This feature is not specified in [GFM][1] specification.

## Examples of (iii) -- Footnotes in markdown on GitHub

GitHub recently added [support for footnotes][8][^1] in markdown on GitHub.

[^1]: This is a footnote.

```markdown
Here is a simple footnote[^1]. With some additional text after it.

[^1]: My reference.
```

This feature is not specified in [GFM][1] specification.

## Options for consideration

To alleviate these interoperability concerns we might want to consider one of the following options.

### Option (A): GFM + HTML::Pipeline rules from 2020

Sticking with what we have, which is consistent with what GitHub was doing around 2020, has some benefits. We know for certain that what works on pub.dev, will also render nicely on GitHub. There is no risk that we accidentally introduce syntax that only works on pub.dev, and doesn't work on GitHub.

Besides it's reasonable to argue that the interoperability issues are really mostly corner cases. And few users are likely to run into these issues.

**Pros:**
* No need to make decisions on whether or not to support every new feature.
* The [GFM][1] specification rarely changes.
* HTML sanitization rules are simple.
* It's easy to have consistency across Dart tooling (pub.dev / dartdoc)
* What works on pub.dev will always work on GitHub.

**Cons:**
* We'll have a few interoperability issues for new GitHub specific features.

### Option (B): GFM + New HTML sanitization rules

We could decide to stick with GFM and adopt new HTML sanitization rules. We can ask GitHub if they are willing to share their rules, or we can adopt our rules as we discover missing features.

**Pros:**
* The [GFM][1] specification rarely changes.
* Incompatibility issues from missing enhancements will degrade gracefully.

**Cons:**
* We might introduce incompatibility issues, if we accidentally allow something that GitHub doesn't allow. Indeed in such senarios might have to do breaking changes.
* We'll still have incompatibility issues in cases of GFM syntax enhancement (i). However, these mostly render reasonably without the syntax enhancements.

This might be a good middle way, and it might be viable for us to implement a few important syntax enhancements on a case-by-case basis, if we really want to. We've certainly talked about making a syntax enhancement for dartpad when it has pub.dev support.

### Option (C): Aim for GitHub interoperability

We could aim for full interoperability, but this might be more than what we need.

**Pros:**
* Users won't experience many interoperability issues.

**Cons:**
* This will require a lot of work and ongoing work (we'll probably need to tweak the markdown parser).
* Some syntax enhancements (i) are not stable yet.
* Shipping support for mermaid across all Dart tooling (pub.dev/dartdoc) might be hard.
* Support for syntax mentioning GitHub users / repositories / issues is a lot of work; and might not make a lot of sense on pub.dev.
* We'll need to decide what GitHub specific syntax enhancements we want to support.

------------------

I'm open to other ideas about how to decide what to support and what not to support. But I'd love for us to establish some guidelines around what we want to support.

For example: I think it's sound if we don't support unstable syntax enhancements like [note/warning blockquotes][5], which is currently in beta.

[1]: https://github.github.com/gfm/
[2]: https://github.blog/2012-11-27-html-pipeline-chainable-content-filters/
[3]: https://github.com/gjtorikian/html-pipeline/pull/329
[4]: https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/
[5]: https://github.com/community/community/discussions/16925
[6]: https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#specifying-the-theme-an-image-is-shown-to
[7]: https://github.blog/changelog/2022-05-19-render-mathematical-expressions-in-markdown/
[8]: https://github.blog/changelog/2021-09-30-footnotes-now-supported-in-markdown-fields/