apache / apache/lucene

How to handle back-compat for new graph TokenFilters? [LUCENE-6721]

Open
#7,779 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

#7722 has a patch for a new synonym filter that correctly handles multi-token synonyms, unlike the known bugs we have today (see http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html for examples).

But since existing query parsers and indexer (and I'm sure many other external analysis consumers) ignore `PosLenAtt`, that patch also has a back-compat layer, `SausageGraphFilter`, to "squash" the graph back down so these components work as best they can...

Anyway, unless we can figure out how to make the back-compat even better than `SausageGraphFilter`, we can't really move forward with graph token filters.

Robert suggested entirely new attributes for graph token streams, but I don't see how that can work: it seems like we'd then need to have 2 copies of certain token filters, e.g. `StopFilter` and `StopGraphFilter`.

Maybe we could fix indexer and at least our query parsers to barf if they every see `PosLenAtt` != 1? Then you'd know you need to add the back compat layer to your analysis chain...

---
Migrated from [LUCENE-6721](https://issues.apache.org/jira/browse/LUCENE-6721) by Michael McCandless (@mikemccand)
Linked issues:
- #7722

Contributor guide

Open the contributing guide

Research direction

Start with linked issue #7722 and its patch, then inspect SausageGraphFilter, PosLenAtt, StopFilter, the indexer, and query parsers. Determine how existing consumers should handle graph token streams and document an agreed compatibility approach; the issue is done when the design is settled well enough to unblock graph token filters.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.