apache / apache/stormcrawler

Remove needsDOM() in ParseFilter

Open
#158 4 comments 0 reactions 0 assignees View on GitHub
good first issue help wanted question
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

This was provided as a way of avoiding building the DOM structure for when no ParseFilters were requiring it and is used only in the Tika parser.

The RobotsTag extractor requires such a DOM so unless we are able to extract the robots directives differently e.g. via SAX and a custom ContentHandler we should reconsider whether this method is really needed.

The JSoupParseFilter could avoid building the DOM too and extract the values via its org.jsoup.nodes.Document prior to building the DOM (if required).

Related to #154

Contributor guide

Open the contributing guide

Research direction

Read ParseFilter and its uses in the Tika parser, then trace how RobotsTag and JSoupParseFilter depend on the DOM. Review related issue #154 before deciding whether needsDOM() can be removed without changing robots-directive extraction. Done means the obsolete method and its callers are removed or revised while the affected parsing behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.