How to handle back-compat for new graph TokenFilters? [LUCENE-6721]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
#7722 has a patch for a new synonym filter that correctly handles multi-token synonyms, unlike the known bugs we have today (see http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html for examples).
But since existing query parsers and indexer (and I'm sure many other external analysis consumers) ignore `PosLenAtt`, that patch also has a back-compat layer, `SausageGraphFilter`, to "squash" the graph back down so these components work as best they can...
Anyway, unless we can figure out how to make the back-compat even better than `SausageGraphFilter`, we can't really move forward with graph token filters.
Robert suggested entirely new attributes for graph token streams, but I don't see how that can work: it seems like we'd then need to have 2 copies of certain token filters, e.g. `StopFilter` and `StopGraphFilter`.
Maybe we could fix indexer and at least our query parsers to barf if they every see `PosLenAtt` != 1? Then you'd know you need to add the back compat layer to your analysis chain...
---
Migrated from [LUCENE-6721](https://issues.apache.org/jira/browse/LUCENE-6721) by Michael McCandless (@mikemccand)
Linked issues:
- #7722
Contributor guide
Research direction
Start with linked issue #7722 and its patch, then inspect SausageGraphFilter, PosLenAtt, StopFilter, the indexer, and query parsers. Determine how existing consumers should handle graph token streams and document an agreed compatibility approach; the issue is done when the design is settled well enough to unblock graph token filters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100