HTML Content Extraction
Open
complexity:high
enhancement
priority:medium
- Dominant language
- Ruby
- Stars
- 224
- Forks
- 48
- Avg merge
- 23h 16m
- Merged PRs (30d)
- 18
Description
### Problem Description
Crawler has the ability to store full pages as HTML, but often only subsets of HTML are useful. For example many sites have key content in xpath(*//main), and current tooling allows us to extract this as text, but not as HTML. Once extracted as HTML, additional work can be undertaken to convert to JSON or Markdown or semantically separated chunks of text.
### Proposed Solution
An option to have content extraction extract content as HTML.
### Alternatives
- Ingestion pipeline elements to handle HTML
- Painless capabilities to handle HTML
- Workarounds outside of platform
Contributor guide
Assessment
This issue has not been assessed yet.