matthewwithanm / matthewwithanm/python-markdownify

Recursion Error: Process_element / process_tag mutual recursion (infinite loop)

Open
#256 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.2k
Forks
203
PR merge metrics
No merged PRs in 30d

Description

When converting HTML that contains circular parent-child references in the parsed BeautifulSoup tree (produced by certain PDF-to-HTML pipelines), process_element and process_tag recurse infinitely, crashing with a RecursionError.

Encountered when using markdownify via marker-pdf to convert PDF documents. The HTML produced by the PDF parser contained structures that caused BeautifulSoup's html.parser to create a non-tree graph where a descendant node held a reference back to an ancestor creating an unbounded call stack.

The mutual recursion introduced in 1.2.2 between process_element and process_tag has no cycle guard:

  • process_tag iterates node.children and calls process_element for each child
  • process_element calls process_tag for any Tag node
RecursionError: maximum recursion depth exceeded

File "markdownify/__init__.py", line 232, in process_element
    return self.process_tag(node, parent_tags=parent_tags)
File "markdownify/__init__.py", line 287, in process_tag
    child_strings = [
File "markdownify/__init__.py", line 288, in <listcomp>
    self.process_element(el, parent_tags=parent_tags_for_children)
File "markdownify/__init__.py", line 232, in process_element
    return self.process_tag(node, parent_tags=parent_tags)
... (repeating until stack exhausted)

Suggested fix:

Pass a visited set of node ids through the call chain to detect and break cycles:

def process_element(self, node, parent_tags=None, _visited=None):
    if isinstance(node, NavigableString):
        return self.process_text(node, parent_tags=parent_tags)
    else:
        return self.process_tag(node, parent_tags=parent_tags, _visited=_visited)

def process_tag(self, node, parent_tags=None, _visited=None):
    if parent_tags is None:
        parent_tags = set()

    # Cycle detection
    if _visited is None:
        _visited = set()
    node_id = id(node)
    if node_id in _visited:
        return ''
    _visited.add(node_id)

    # ... rest of method unchanged, but pass _visited= to process_element calls
    child_strings = [
        self.process_element(el, parent_tags=parent_tags_for_children, _visited=_visited)
        for el in children_to_convert
    ]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in markdownify/init.py by reading process_element and process_tag, then reproduce the RecursionError with an HTML tree containing a circular reference. Trace how child processing passes through both methods and verify that conversion terminates without exhausting the stack while normal HTML conversion remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.