apache / apache/lucene

HTMLStripCharFilter can not remove html tags

Open
#14,089 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

`

public AnalyzerResult analyze(String text) throws IOException {
text = HtmlExtractor.extractTextFromHtml(text);
List tokens = new ArrayList<>();
List originalTexts = new ArrayList<>();
try (TokenStream stream = tokenStream("*", text)) {
stream.reset();
CharTermAttribute charTermAttribute = stream.addAttribute(CharTermAttribute.class);
OffsetAttribute offsetAttribute = stream.addAttribute(OffsetAttribute.class);
while (stream.incrementToken()) {
tokens.add(charTermAttribute.toString());
originalTexts.add(text.substring(offsetAttribute.startOffset(), offsetAttribute.endOffset()));
}
}
return AnalyzerResult.builder().tokens(tokens).originalTexts(originalTexts).build();
}
`

`
public static String extractTextFromHtml(String content) {
Document document = Jsoup.parseBodyFragment(content);
return document.body().text().replace(" ", "").trim();
}
`

`
protected Reader initReader(String fieldName, Reader reader) {
reader = new HTMLStripCharFilter(reader);
reader = new JapaneseIterationMarkCharFilter(reader);
return reader;
}
`

`
@PostConstruct
@Scheduled(cron = "0 0 0 * * *")
public void init() {
Logged.L.info("load Japanese config.");
String dict = japaneseDictConfig.getDict();
UserDictionary userDictionary = null;
try {
userDictionary = UserDictionary.open(new StringReader(dict));
} catch (Exception e) {
Logged.L.error("load japanese dict error", e);
}

List stopWords = stopWordConfig.getStopWords();
CharArraySet stopSet = new CharArraySet(stopWords, true);
stopSet.add(getDefaultStopSet());

Tokenizer tokenizer = new JapaneseTokenizer(userDictionary, true, false, JapaneseTokenizer.Mode.SEARCH);

TokenStream stream = new JapaneseBaseFormFilter(tokenizer);
stream = new JapanesePartOfSpeechStopFilter(stream, getDefaultStopTags());
stream = new CJKWidthFilter(stream);
stream = new StopFilter(stream, stopSet);
stream = new JapaneseReadingFormFilter(stream);
// stream = new JapaneseKatakanaStemFilter(stream);
stream = new JapaneseNumberFilter(stream);
stream = new LowerCaseFilter(stream);
this.tokenStreamComponents = new TokenStreamComponents(tokenizer, stream);
}
`
`@Test
public void test() throws SQLException, IOException {
String ss = "背景";
MultiLanguageAnalyzer.AnalyzerResult analyzerResult = analyzer.analyze(ss);
System.out.println(analyzerResult.getOriginalTexts());

}`
The code is like this. I use lucene-analyzers kuromoji 8.11.4
If I do not filter html using jsoup, The output originalTexts will be `背景`. The html will still exist, does this result match the expectation?

If I use jsoup to filter the input first, the output will be `背景`,

### Version and environment details

org.apache.lucene
lucene-analyzers-kuromoji
8.11.4

Contributor guide

Open the contributing guide

Research direction

Start with the provided analyzer test and trace the reader chain through HTMLStripCharFilter, JapaneseIterationMarkCharFilter, and JapaneseTokenizer in Lucene 8.11.4. Compare the token offsets and originalTexts for the span example with the documented HTMLStripCharFilter behavior; done means the expected behavior is established and any discrepancy is reproduced in a focused test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.