[Bug] NOT_IMPLEMENTED_ERROR: Custom analyzer is not supported for unindexed MATCH operations when using match_phrase with OR subquery
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
2. Steps to Reproduce (DDL & Indexes)
SQL
-- 1. Create Custom Tokenizer
CREATE INVERTED INDEX TOKENIZER IF NOT EXISTS ngram_2_tokenizer
PROPERTIES (
"type" = "ngram",
"min_gram" = "2",
"max_gram" = "2"
);
-- 2. Create Analyzer with lowercase filter
CREATE INVERTED INDEX ANALYZER ngram_2_analyzer_lc
PROPERTIES (
"tokenizer" = "ngram_2_tokenizer",
"token_filter" = "lowercase"
);
-- 3. Add Inverted Indexes to existing table
ALTER TABLE pdm_index_table
ADD INDEX idx_name_ngram2(name)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
ALTER TABLE pdm_index_table
ADD INDEX idx_number_ngram2(number)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
ALTER TABLE xdm_xdmfilecontent
ADD INDEX idx_content_ngram2(attachment_content)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
-- 4.build index
BUILD INDEX idx_name_ngram2 ON pdm_index_table;
BUILD INDEX idx_number_ngram2 ON pdm_index_table;
BUILD INDEX idx_content_ngram2 ON xdm_xdmfilecontent;
3. Comparison: What Works vs. What Fails
Case 1: Same-table OR (WORKS ✅)
● SQL
-- This query runs successfully using inverted index
-- success running 1
SELECT id FROM pdm_index_table
WHERE (name MATCH_PHRASE 'ghm' OR number MATCH_PHRASE 'ghm')
AND workingState <> 'CHECKED_OUT';
-- success running 2
select IndexA.id as 'IndexA_id'
from plm20_sit_xdm_f.pdm_index_table as IndexA
left join plm20_sit_xdm_f.pdm_index_master_table as IndexA$master on IndexA$master.id = IndexA._masterid
where (IndexA.className in ('Document'))
and ((IndexA.name match_phrase 'ghm' or
IndexA.number match_phrase 'ghm' )
and IndexA$master.isTemplate = 0)
and (IndexA.workingState <> 'CHECKED_OUT')
union all
select IndexA.id as 'IndexA_id'
from plm20_sit_xdm_f.pdm_index_table as IndexA
left join plm20_sit_xdm_f.pdm_index_master_table as IndexA$master on IndexA$master.id = IndexA._masterid
where (IndexA.className in ('Document'))
and ((
IndexA.id in (select SubSelect1FileContent.id as 'SubSelect1FileContent_id'
from plm20_sit_xdm_f.xdm_xdmfilecontent as SubSelect1FileContent
where SubSelect1FileContent.attachment_content match_phrase 'ghm' ))
and IndexA$master.isTemplate = 0)
and (IndexA.workingState <> 'CHECKED_OUT');
Case 2: OR with Subquery (FAILS ❌)
● SQL
-- This query triggers [NOT_IMPLEMENTED_ERROR]
SELECT IndexA.id
FROM pdm_index_table as IndexA
WHERE (
IndexA.number MATCH_PHRASE 'ghm'
OR IndexA.id IN (
SELECT SubContent.id
FROM xdm_xdmfilecontent as SubContent
WHERE SubContent.attachment_content MATCH_PHRASE 'ghm'
)
)
AND IndexA.workingState <> 'CHECKED_OUT';
4. Error MessageerrCode = 2, detailMessage = (192.168.10.235)[NOT_IMPLEMENTED_ERROR]Custom analyzer is not supported for unindexed MATCH operations.
5. Analysis For Case 1, Doris performs a Bitmap OR at the storage layer. However, for Case 2, the OR logic with a Semi-Join (Subquery) causes the match_phrase predicate to be hoisted to the VSelectNode (computational layer) instead of being pushed down to the VOlapScanNode. Since match_phrase with a custom analyzer requires an inverted index to function, it throws an error when evaluated as a standard expression in an unindexed path.
### What's Wrong?
2. Steps to Reproduce (DDL & Indexes)
SQL
-- 1. Create Custom Tokenizer
CREATE INVERTED INDEX TOKENIZER IF NOT EXISTS ngram_2_tokenizer
PROPERTIES (
"type" = "ngram",
"min_gram" = "2",
"max_gram" = "2"
);
-- 2. Create Analyzer with lowercase filter
CREATE INVERTED INDEX ANALYZER ngram_2_analyzer_lc
PROPERTIES (
"tokenizer" = "ngram_2_tokenizer",
"token_filter" = "lowercase"
);
-- 3. Add Inverted Indexes to existing table
ALTER TABLE pdm_index_table
ADD INDEX idx_name_ngram2(name)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
ALTER TABLE pdm_index_table
ADD INDEX idx_number_ngram2(number)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
ALTER TABLE xdm_xdmfilecontent
ADD INDEX idx_content_ngram2(attachment_content)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
-- 4.build index
BUILD INDEX idx_name_ngram2 ON pdm_index_table;
BUILD INDEX idx_number_ngram2 ON pdm_index_table;
BUILD INDEX idx_content_ngram2 ON xdm_xdmfilecontent;
3. Comparison: What Works vs. What Fails
Case 1: Same-table OR (WORKS ✅)
● SQL
-- This query runs successfully using inverted index
-- success running 1
SELECT id FROM pdm_index_table
WHERE (name MATCH_PHRASE 'ghm' OR number MATCH_PHRASE 'ghm')
AND workingState <> 'CHECKED_OUT';
-- success running 2
select IndexA.id as 'IndexA_id'
from plm20_sit_xdm_f.pdm_index_table as IndexA
left join plm20_sit_xdm_f.pdm_index_master_table as IndexA$master on IndexA$master.id = IndexA._masterid
where (IndexA.className in ('Document'))
and ((IndexA.name match_phrase 'ghm' or
IndexA.number match_phrase 'ghm' )
and IndexA$master.isTemplate = 0)
and (IndexA.workingState <> 'CHECKED_OUT')
union all
select IndexA.id as 'IndexA_id'
from plm20_sit_xdm_f.pdm_index_table as IndexA
left join plm20_sit_xdm_f.pdm_index_master_table as IndexA$master on IndexA$master.id = IndexA._masterid
where (IndexA.className in ('Document'))
and ((
IndexA.id in (select SubSelect1FileContent.id as 'SubSelect1FileContent_id'
from plm20_sit_xdm_f.xdm_xdmfilecontent as SubSelect1FileContent
where SubSelect1FileContent.attachment_content match_phrase 'ghm' ))
and IndexA$master.isTemplate = 0)
and (IndexA.workingState <> 'CHECKED_OUT');
Case 2: OR with Subquery (FAILS ❌)
● SQL
-- This query triggers [NOT_IMPLEMENTED_ERROR]
SELECT IndexA.id
FROM pdm_index_table as IndexA
WHERE (
IndexA.number MATCH_PHRASE 'ghm'
OR IndexA.id IN (
SELECT SubContent.id
FROM xdm_xdmfilecontent as SubContent
WHERE SubContent.attachment_content MATCH_PHRASE 'ghm'
)
)
AND IndexA.workingState <> 'CHECKED_OUT';
4. Error MessageerrCode = 2, detailMessage = (192.168.10.235)[NOT_IMPLEMENTED_ERROR]Custom analyzer is not supported for unindexed MATCH operations.
5. Analysis For Case 1, Doris performs a Bitmap OR at the storage layer. However, for Case 2, the OR logic with a Semi-Join (Subquery) causes the match_phrase predicate to be hoisted to the VSelectNode (computational layer) instead of being pushed down to the VOlapScanNode. Since match_phrase with a custom analyzer requires an inverted index to function, it throws an error when evaluated as a standard expression in an unindexed path.
### What You Expected?
Expected Behavior
The query optimizer should be able to either:
Push down the match_phrase part to the inverted index separately.
Or handle the OR logic without breaking the indexed search path.
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.