The commonmark renderer's escaping strategy is very aggresive

I am currently evaluating the usefulness of cmark as an indentation / formatting tool.

The width parameter makes cmark already quite suitable, though I would remove all soft breaks and normalize before rendering in case HARDBREAKS wasn't passed, this would avoid such cases:

A very long line that the user manually broke because it was really getting
too long

rendering to

A very long line that the user manually broke because it was
really getting
too long

Anyway, my real issue here is that a lot of characters are preemptively escaped, even though they could actually pass through unescaped. I understand that the initial intent was (and rightly so) "better safe than sorry", but I think we should now look back at how to be more clever wrt these characters, starting with the most likely to be used in a common sentence.

Case in point: !

Currently given that input:

Hello amigo!

we have this output when running cat hello.md | ./build/src/cmark -t commonmark

Hello amigo\!

This was introduced by efeb7093a3e6a96019b0805fc630a7aa4c31481b. The specific reason for this is not stated in the commit message, but when resetting cmark to the previous commit, and running make roundtrip_test , it becomes clear:

Example 533 (lines 7317-7323) Images
\![foo]

[foo]: /url "title"

--- expected HTML
+++ actual HTML
@@ -1 +1 @@
-<p>!<a href="/url" title="title">foo</a></p>
+<p><img src="/url" alt="foo" title="title" /></p>

589 passed, 1 failed, 0 errored, 0 skipped

I haven't investigated why, but I suspect the roundtrip tests are broken in current master,
as even when deactivating all escaping in the renderer, the roundtrip_executable test still happily
reports success. The issue can be reproduced with current master by "reverting" this
commit (might conflict, edited the case out) and running rountrip.sh manually.

Furthermore, some code was obsoleted in this commit (but still lives in the current HEAD):

needs_escaping == [...] c == '!' || [...] || (c == '!' && nextc == '[')

This should be removed, but helps us understand the issue at hand, and the shortcomings of the current strategy:

The previous code tried to determine whether ! should be escaped by looking at the next character in the literal stream it was parsed from, bounded by its containing block limits. However here, the next character is NULL, and the issue was triggered by the next character we output, which was '['.

The fix that was introduced for this escaped all ! , end of story. However I think a more appropriate fix would be to escape it "retroactively": the only case I'm aware of where '!' can have a semantic meaning is when it is followed by '[', as such we should, in the case where the renderer ends up outputting it on its own (case CMARK_NODE_LINK), be able to verify that the previous character that was output was a "!", and only then insert an escape character in the strbuf. We should obviously make sure to escape it only if it was contained in a TEXT node, but that shouldn't be too hard.

I haven't studied the code further yet, because I thought this was the most glaring issue, however I also strongly dislike the underscore escaping, because it requires function_names to be wrapped in a code inline in order to escape escaping.

We should determine the conditions under which most if not all of these characters need to be escaped, and try to determine if we can come up with a better strategy, in order to make cmark more usable as a source-to-source tool, I hope you folks can help me study each case :)

Start with the commonmark renderer's escaping logic and run make roundtrip_test plus roundtrip.sh, using the ! case and Example 533 as regression points. Compare build/src/cmark output with the expected HTML, then define a narrower escaping strategy for text and link rendering; done means ordinary punctuation is not unnecessarily escaped while link and image round trips still pass.