apache / apache/lucene

CachingTokenFilter can cause close() to be called twice. [LUCENE-3089]

Open
#4,162 11 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In #4137, we added some state and checks to MockTokenizer to validate that consumers
are properly using the tokenstream workflow (described here: http://lucene.apache.org/java/3_0_3/api/core/org/apache/lucene/analysis/TokenStream.html)

One problem I noticed in TestTermVectorsWriter.testEndOffsetPositionWithCachingTokenFilter is that providing a CachingTOkenFilter directly will result
in close() being called twice on the underlying tokenstream... this seems wrong.

Some ideas to fix this could be:
1. CachingTokenFilter overrides close() and we document that you must close the underlying stream yourself. I think this is what the queryparser does anyway.
1. CachingTokenFilter does something tricky to ensure it only closes the underlying stream once.

---
Migrated from [LUCENE-3089](https://issues.apache.org/jira/browse/LUCENE-3089) by Robert Muir (@rmuir), updated Nov 14 2011

Contributor guide

Open the contributing guide

Research direction

Start with TestTermVectorsWriter.testEndOffsetPositionWithCachingTokenFilter and inspect how CachingTokenFilter, MockTokenizer, and the token-stream workflow handle close(). Review the queryparser usage mentioned in the issue, then add coverage for the chosen close behavior and verify that the underlying token stream is not closed twice.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.