Unicode escape no longer works for non-suffix-only wildcard terms [LUCENE-4382]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
#1666 added support for escaping of wildcard characters, but when the de-escaping logic was pushed down from the query parser (QueryParserBase) into WildcardQuery, support for Unicode escaping (backslash, "u", and the four-digit hex Unicode code) was not included.
Two solutions:
1. Do the Unicode de-escaping in the query parser before calling getWildcardQuery.
2. Support Unicode de-escaping in WildcardQuery.
A suffix-only wildcard does not exhibit this problem because full de-escaping is performed in the query parser before calling getPrefixQuery.
My test case, added at the beginning of TestExtendedDismaxParser.testFocusQueryParser:
```Java
assertQ("expected doc is missing (using escaped edismax w/field)",
req("q", "t_special:literal\\:\\u0063olo*n",
"defType", "edismax"),
"//doc[1]/str[@name='id'][.='46']");
```
Note: That test case was only used to debug into WildcardQuery to see that the Unicode escape was not processed correctly. It fails in all cases, but that's because of how the field type is analyzed.
Here is a Lucene-level test case that can also be debugged to see that WildcardQuery is not processing the Unicode escape properly. I added it at the start of TestMultiAnalyzer.testMultiAnalyzer:
```Java
assertEquals("literal\\:\\u0063olo*n", qp.parse("literal\\:\\u0063olo*n").toString());
```
Note: This case will always run correctly since it is only checking the input pattern string for WildcardQuery and not how the de-escaping was performed within WildcardQuery.
---
Migrated from [LUCENE-4382](https://issues.apache.org/jira/browse/LUCENE-4382) by Jack Krupansky, updated May 09 2016
Contributor guide
Research direction
Start with QueryParserBase and WildcardQuery to trace where Unicode de-escaping is currently handled. Review the cases in TestExtendedDismaxParser.testFocusQueryParser and TestMultiAnalyzer.testMultiAnalyzer, then determine which proposed approach fits the existing flow. Done means non-suffix-only wildcard terms correctly process backslash-u four-digit Unicode escapes without regressing suffix-only wildcard behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100