Preserve uncommon-but-real Markdown constructs verbatim (don't reflow what the parser doesn't model)

# Preserve uncommon-but-real Markdown constructs verbatim (don't reflow what the parser doesn't model)

## Summary

flowmark parses to a comrak AST and **re-emits + reflows** (semantic line breaks, width wrap). Any construct comrak doesn't model as an opaque node is therefore treated as prose and **reflowed — often corrupting it**. Because flowmark is designed to be "safe to run automatically on save or at any stage of a document pipeline," the correct goal here is **preservation**: round-trip these constructs **verbatim**, even when flowmark can't (and shouldn't) fully *parse* the more unusual variants (e.g. `::: tab Title`). A formatter must never silently corrupt input it doesn't understand.

This isn't a request to adopt or parse any one dialect — there is **no single standard** for several of these (see below). It's a request to **never mangle** them.

> Applies to both **jlevy/flowmark** (Python) and **jlevy/flowmark-rs** (the comrak-based Rust port); the empirical findings below were verified against the Rust port's source + the released `flowmark 0.3.1` binary.

## flowmark already does this — for some constructs

The mechanism already exists and is battle-tested. `flowmark-rs/src/formatter/filling.rs` documents a pre-parse **PUA-marker passthrough (COMRAK-WORKAROUND1–12)** that already protects: reference links, footnote definitions, autolink angle brackets, backslash escapes, HTML tags, and **Jinja/Markdoc/Liquid/Hugo template tags** (`{{ }}`, `{% %}`, `{{< >}}`); YAML `---` frontmatter is handled out-of-band in `parser/frontmatter.rs`. The README states the conservative-preservation philosophy, and there's an inline `atomic_spans` tokenizer (code spans, links, autolinks, bare URLs, HTML/Jinja tags).

**The gap is which constructs are wired into that passthrough — and that it's currently inline-level; several corruptions are block-level.** This issue is to extend the same "treat as opaque" guarantee to the remaining common constructs, at the block level where needed.

## The corruption ledger (empirically verified against `flowmark 0.3.1`)

Each row was observed directly via a round-trip battery (input → `flowmark` → output).

| Construct | Example | Observed result | Severity |
|---|---|---|---|
| **Pandoc multiline tables** | dashed-rule blocks | **Catastrophic**: dashed rules became `* * *` thematic breaks + `##` headings; rows merged | 🔴 P0 |
| **Obsidian callouts** | `> [!tip]+ My Title` | → `> [!TIP]` — **drops the custom title AND the fold `+`** (real data loss) | 🔴 P0 |
| **markdown-it-container / Pandoc fenced divs** | `::: warning … :::`, `::: {.note}` | collapsed to prose (panels/blocks merged) | 🔴 P0 |
| **TOML / `+++` frontmatter** | `+++ … +++` | reflowed into one prose line (`split_frontmatter` only knows `---`) | 🔴 P0 |
| **Definition lists** | `Term`⏎`: Def` | → `Term : Def` (collapsed; comrak `description_lists` is OFF) | 🔴 P0 |
| **`\[…\]` block math / `$$…$$`** | `\[ … \]`, `$$ … $$` | collapsed onto one line (`\[ \]` not a comrak math delimiter; `$$` block reflowed) | 🟠 P1 |
| **Pandoc grid tables** | `+---+` / `|===|` | not recognized; body cells reflow | 🟠 P1 |
| **Raw multi-line HTML blocks** | `

` | collapsed onto one line (valid HTML, but structure/diffs destroyed) | 🟠 P1 |
| **`{.class #id key=val}` attribute lists/spans** | `[text]{.mark}`, `# H {.c}` | no Rust parser models them (comrak/pulldown-cmark/markdown-rs); reflowed/splittable | 🟠 P1 |
| **Line blocks (`|`)** | `| verse line` | a lone `|` line reflows and loses the `|` (survives only incidentally via the table-row heuristic) | 🟡 P2 |
| **LaTeX `\(…\)` inline** | `\(x^2\)` | reflowed as prose (comrak math is `$`-only); survives only by luck | 🟡 P2 |
| **MyST roles `` {role}`…` ``, wikilinks `[[Page]]`** | `` {sub}`x` ``, `[[Page]]` | survive only when an adjacent atomic span happens to cover them | 🟡 P2 |

**Already robustly safe (no action needed):** GFM pipe tables, GFM alerts (`> [!NOTE]`), task lists, footnote refs/defs, strikethrough, fenced/indented code, autolinks, reference links/defs, inline HTML tags, **Hugo `{{< >}}` + Jekyll `{% %}` shortcodes** (protected by the existing template-tag patterns), YAML `---` frontmatter, HTML entities, and the interior of `$…$` inline math. (These are backed either by an enabled comrak extension or by an existing WORKAROUND — that's exactly why they're safe, and the model to follow.)

## Prioritized fixes

**P0 — real data loss / catastrophic mangling**
1. Pandoc **multiline tables** (worst — produces wrong block types).
2. **Obsidian callouts** (silent loss of custom title + fold marker; very common format).
3. **`:::` containers + Pandoc fenced divs** (collapses blocks/panels).
4. **TOML/`+++` frontmatter** (small fix: teach `split_frontmatter` the `+++` delimiter, mirroring `---`).
5. **Definition lists** (enable + test comrak `description_lists`, or passthrough).

**P1 — valid output but destroyed structure**
6. `\[…\]` / `$$…$$` block math (add `\[ \]`/`\( \)` to math protection; don't reflow `$$` blocks).
7. Pandoc **grid tables** (opaque `^[+|]` block).
8. **Raw multi-line HTML blocks** (preserve internal line structure).
9. **`{.class}` attribute lists/spans** (no Rust parser support — needs the pre-parse passthrough).

**P2 — harden the "safe only by accident" cases**
10. **Line blocks (`|`)** as a first-class opaque block (don't rely on the table-row heuristic).
11. **MyST roles / wikilinks** explicit atomic patterns.
12. **`\(…\)`** folded into the math-protection pass.

## On comrak 0.52

The dependabot bump (flowmark-rs #73) to comrak 0.52 adds a `block_directive` extension — but it covers **`:::` only**, doesn't handle `::: {.class}` info-string/nesting semantics like markdown-it-container, and comrak **still has no fields** for attributes, fenced divs (Pandoc-style), grid tables, Pandoc definition lists, line blocks, or TOML frontmatter. So a comrak upgrade alone won't fix most of the above. (comrak's `shortcodes` is emoji `:smile:`, unrelated.)

## Why "no single standard" → preserve, don't parse

CommonMark lists both **Directives** and **Attributes** only as *proposed* extensions. `:::` has at least four mutually-incompatible dialects — Pandoc `fenced_divs`, markdown-it-container (info strings, e.g. `::: tab Title`), CommonMark generic-directives / remark-directive / MyST (`:::{note}`), MyST `colon_fence` — and even the nearest convergence (remark-directive) warns against relying on it in pipelines you don't control. `{.class}` likewise has incompatible Pandoc vs markdown-it-attrs forms. There is no correct "parse it" — only "don't break it."

## Suggested approach

Extend the existing pre-parse PUA-marker passthrough (the WORKAROUND1–12 mechanism) to the constructs above — preferring **parser-independent, dialect-agnostic** rules so they're robust across flavors:

- **`:::` fenced regions:** treat `^:{3,}` opener → bare `:::`-run closer (count need not match, per Pandoc) as an **atomic block**; don't reflow the fences or inner lines. Covers all four dialects without parsing any info string.
- **`{.class}` lines/spans:** protect a line that is only a brace attribute group (`^\s*\{[.#][^}]*\}\s*$`) and inline `]{…}` spans.
- **Block math:** add `\[…\]` / `\(…\)` to the math protection and stop reflowing `$$` blocks.
- **`+++` frontmatter:** add the delimiter to `split_frontmatter`.
- **Callouts / multiline+grid tables / def lists / raw HTML blocks / line blocks:** opaque-block passthrough (or, where a comrak extension exists and round-trips, enable + test it).

This is an extension of a proven mechanism, not new infrastructure — and it's more robust than waiting on comrak, which still lacks fields for most of these.

Key files (flowmark-rs): `src/formatter/filling.rs` (passthrough + workaround docs), `src/wrapping/atomic_patterns.rs`, `src/wrapping/tag_handling.rs`, `src/parser/frontmatter.rs`, `src/formatter/markdown.rs` (extension toggles).

## Sources

- CommonMark Proposed Extensions: https://github.com/commonmark/commonmark-spec/wiki/Proposed-Extensions
- Pandoc Divs/Spans + Tables + Line blocks + Definition lists: https://pandoc.org/MANUAL.html , https://pandoc.org/demo/example33/8.18-divs-and-spans.html , https://pandoc.org/demo/example33/8.6-line-blocks.html
- markdown-it-container: https://github.com/markdown-it/markdown-it-container • tab plugin: https://mdit-plugins.github.io/tab.html • markdown-it-attrs: https://github.com/arve0/markdown-it-attrs
- remark-directive: https://github.com/remarkjs/remark-directive • generic-directives proposal: https://talk.commonmark.org/t/generic-directives-plugins-syntax/444 • MyST optional syntax: https://myst-parser.readthedocs.io/en/latest/syntax/optional.html
- comrak `block_directive` + extension fields: https://docs.rs/comrak/latest/comrak/options/struct.Extension.html • releases: https://github.com/kivikakk/comrak/releases
- GitHub alerts: https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax • Obsidian callouts: https://help.obsidian.md/Editing+and+formatting/Callouts
- MathJax delimiters (`\(`,`\[` are the defaults, not `$`): https://docs.mathjax.org/en/latest/input/tex/delimiters.html
- Prior-art corruption: prettier ignoring directives https://github.com/prettier/prettier/issues/19040 ; GFM alerts https://github.com/prettier/prettier/issues/15479 ; mdformat plugin round-trip precedent https://github.com/hukkin/mdformat

*Findings verified by inspecting flowmark-rs source (comrak 0.51 pinned) and running a round-trip battery against `flowmark 0.3.1`.*

Start with flowmark-rs/src/formatter/filling.rs and review the existing PUA-marker passthrough and WORKAROUND documentation, then inspect src/wrapping/atomic_patterns.rs, src/wrapping/tag_handling.rs, src/parser/frontmatter.rs, and src/formatter/markdown.rs. Run the reported round-trip cases and preserve each listed unsupported construct verbatim without regressing the constructs already marked safe.

Written by the indexing model from the issue text.