elastic / elastic/crawler

HTML Content Extraction

Open
#144 9 comments 0 reactions 0 assignees View on GitHub
complexity:high enhancement priority:medium
Dominant language
Ruby
Stars
224
Forks
48
Avg merge
23h 16m
Merged PRs (30d)
18

Description

### Problem Description
Crawler has the ability to store full pages as HTML, but often only subsets of HTML are useful. For example many sites have key content in xpath(*//main), and current tooling allows us to extract this as text, but not as HTML. Once extracted as HTML, additional work can be undertaken to convert to JSON or Markdown or semantically separated chunks of text.

### Proposed Solution
An option to have content extraction extract content as HTML.

### Alternatives
- Ingestion pipeline elements to handle HTML
- Painless capabilities to handle HTML
- Workarounds outside of platform

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.