apache / apache/lucene

Custom attributes get cleared by SynonymFilter [LUCENE-6667]

Open
#7,725 9 comments 0 reactions 0 assignees View on GitHub
affects-version:4.10.4 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I believe the Lucene API enables users to define their custom attributes (by extending `AttributeImpl`) which may be added by custom Tokenizers.
It seems, the `clear` and `copyTo` methods must be implemented to clear and restore the state of this custom attribute.

However, some filters (in our case the SynonymFilter) simply call `AttributeSource.clearAttributes` without invoking `copyTo`. Instead the filter just resets some known attributes, simply ignoring all other custom attributes. In the end our custom attribute value is lost.

Is this a bug in `SynonymFilter` (and others) or are we using the API in the wrong way?

A solution might be of course to provide empty implementations of `clear` and `copyTo`, but I'm not sure if this has other unwanted effects.

---
Migrated from [LUCENE-6667](https://issues.apache.org/jira/browse/LUCENE-6667) by Oliver Becker

Contributor guide

Open the contributing guide

Research direction

Start by tracing SynonymFilter's use of AttributeSource.clearAttributes and compare it with the custom AttributeImpl clear and copyTo contract described in the issue. Reproduce the loss of a custom attribute through the filter, then confirm the API behavior and add coverage if the intended behavior is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.