Unstructured-IO / Unstructured-IO/unstructured
Large HTML documents cannot be partitioned using the partition_html function.
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 15.5k
- Forks
- 1.3k
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 13
Description
partitioning large html documents leads to an empty result. This is due to the missing huge_tree option on HTML parser generation in:
https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/html/parser.py#L929
Fix:
Include the huge_tree option to solve this: etree.HTMLParser(remove_comments=True, huge_tree=True)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in unstructured/partition/html/parser.py at the HTMLParser construction around line 929, then trace how partition_html handles a large document. Confirm that the parser uses the requested huge-tree setting and verify that partitioning a large HTML document no longer returns an empty result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- html
- Domain
- data
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100