commonmark / commonmark/cmark

Illegal control characters in XML output

Open
#365 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
2k
Forks
691
Avg merge
1d 16h
Merged PRs (30d)
1

Description

Hi, I maintain the R bindings for cmark. One popular use case is converting commonmark to xml for processing the AST.

We are running into a problem when input markdown contains control characters (often captured from a tty), which makes xml output invalid. For example if the markdown text contains \033 and we convert that to xml, we get:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
  <paragraph>
    <text xml:space="preserve"></text>
  </paragraph>
</document>

However, trying to parse this with libxml2 fails:

 Error in read_xml.raw(charToRaw(enc2utf8(x)), "UTF-8", ..., as_html = as_html,  : 
  PCDATA invalid Char value 27 [9] 

A real world example is this readme file. This was done with the gfm fork, but I think the problem appears the same.

Is this a bug in cmark, or is markdown text not supposed to contain c0 characters in the first place?

cc @nwellnhof

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names no source file, test, or entry point. Start by reproducing the shown control-character input through the XML renderer, then locate the renderer and its output tests; determine the expected handling and verify that the resulting XML is accepted by libxml2.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.