docling-project / docling-project/docling

Adding Header Grouping to Markdown Backend

Open
#1,893 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

Howdy, grouping under headers is not provided out-of-the-box by Marko so the docling markdown backend does not group elements under headings.

It looks like it would not be hard to provided a custom parser to Marko with a change that would nest content under corresponding headers.

I have a small POC working which does it like this (with minimal changes to md_backend.py)

```
class GroupingParser(Parser):
r"""
All elements defined in CommonMark's spec are included in the parser
by default.

Attributes:
block_elements(dict): a dict of name: block_element pairs
inline_elements(dict): a dict of name: inline_element pairs

:param \*extras: extra elements to be included in parsing process.
"""

@override
def parse_source(self, source: Source) -> list[block.BlockElement]:
"""Parse the source into a list of block elements."""
element_list = self._build_block_element_list()

headings: list[block.Heading | None] = []

ast: list[block.BlockElement] = []
while not source.exhausted:
for ele_type in element_list:
if ele_type.match(source):
result = ele_type.parse(source) # pyright: ignore[reportAny]

if not hasattr(result, "priority"): # pyright: ignore[reportAny]
# In some cases ``parse()`` won't return the element, but
# instead some information to create one, which will be passed
# to ``__init__()``.
result = ele_type(result) # type: ignore # pyright: ignore[reportCallIssue]

if isinstance(result, block.Heading):
headings = insert_heading(headings=headings, new_heading=result)

parent_heading = get_parent_heading(headings=headings)
if parent_heading:
_ = add_children(parent=parent_heading, child=result)
else:
ast.append(result)

raw_text = result.inline_body
if raw_text:
# new_result = RawText(match=f"{"#" * result.level} {raw_text}")
new_result = RawText(match=raw_text)
blank_line = BlankLine(start=0)
result.inline_body = ""
result.children = [new_result, blank_line]

elif headings and headings[-1]:
current_children = headings[-1].children
headings[-1].children = [*current_children, result]
else:
ast.append(result)
break
else:
# Quit the current parsing and go back to the last level.
break
return ast
```

It may be possible to do this in less code by simply wrapping parse source instead of overriding/reimplementing it, I'll check that out a bit later

Is this something that would be worth cleaning up and putting into a PR?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.