apache / apache/doris

[Bug] NOT_IMPLEMENTED_ERROR: Custom analyzer is not supported for unindexed MATCH operations when using match_phrase with OR subquery

Open
#62,170 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

2. Steps to Reproduce (DDL & Indexes)
SQL
-- 1. Create Custom Tokenizer
CREATE INVERTED INDEX TOKENIZER IF NOT EXISTS ngram_2_tokenizer
PROPERTIES (
"type" = "ngram",
"min_gram" = "2",
"max_gram" = "2"
);

-- 2. Create Analyzer with lowercase filter
CREATE INVERTED INDEX ANALYZER ngram_2_analyzer_lc
PROPERTIES (
"tokenizer" = "ngram_2_tokenizer",
"token_filter" = "lowercase"
);

-- 3. Add Inverted Indexes to existing table
ALTER TABLE pdm_index_table
ADD INDEX idx_name_ngram2(name)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);

ALTER TABLE pdm_index_table
ADD INDEX idx_number_ngram2(number)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
ALTER TABLE xdm_xdmfilecontent
ADD INDEX idx_content_ngram2(attachment_content)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
-- 4.build index
BUILD INDEX idx_name_ngram2 ON pdm_index_table;
BUILD INDEX idx_number_ngram2 ON pdm_index_table;
BUILD INDEX idx_content_ngram2 ON xdm_xdmfilecontent;
3. Comparison: What Works vs. What Fails
Case 1: Same-table OR (WORKS ✅)
● SQL
-- This query runs successfully using inverted index
-- success running 1
SELECT id FROM pdm_index_table
WHERE (name MATCH_PHRASE 'ghm' OR number MATCH_PHRASE 'ghm')
AND workingState <> 'CHECKED_OUT';
-- success running 2
select IndexA.id as 'IndexA_id'
from plm20_sit_xdm_f.pdm_index_table as IndexA
left join plm20_sit_xdm_f.pdm_index_master_table as IndexA$master on IndexA$master.id = IndexA._masterid
where (IndexA.className in ('Document'))
and ((IndexA.name match_phrase 'ghm' or
IndexA.number match_phrase 'ghm' )
and IndexA$master.isTemplate = 0)
and (IndexA.workingState <> 'CHECKED_OUT')
union all
select IndexA.id as 'IndexA_id'
from plm20_sit_xdm_f.pdm_index_table as IndexA
left join plm20_sit_xdm_f.pdm_index_master_table as IndexA$master on IndexA$master.id = IndexA._masterid
where (IndexA.className in ('Document'))
and ((
IndexA.id in (select SubSelect1FileContent.id as 'SubSelect1FileContent_id'
from plm20_sit_xdm_f.xdm_xdmfilecontent as SubSelect1FileContent
where SubSelect1FileContent.attachment_content match_phrase 'ghm' ))
and IndexA$master.isTemplate = 0)
and (IndexA.workingState <> 'CHECKED_OUT');

Case 2: OR with Subquery (FAILS ❌)
● SQL
-- This query triggers [NOT_IMPLEMENTED_ERROR]
SELECT IndexA.id
FROM pdm_index_table as IndexA
WHERE (
IndexA.number MATCH_PHRASE 'ghm'
OR IndexA.id IN (
SELECT SubContent.id
FROM xdm_xdmfilecontent as SubContent
WHERE SubContent.attachment_content MATCH_PHRASE 'ghm'
)
)
AND IndexA.workingState <> 'CHECKED_OUT';
4. Error MessageerrCode = 2, detailMessage = (192.168.10.235)[NOT_IMPLEMENTED_ERROR]Custom analyzer is not supported for unindexed MATCH operations.
5. Analysis For Case 1, Doris performs a Bitmap OR at the storage layer. However, for Case 2, the OR logic with a Semi-Join (Subquery) causes the match_phrase predicate to be hoisted to the VSelectNode (computational layer) instead of being pushed down to the VOlapScanNode. Since match_phrase with a custom analyzer requires an inverted index to function, it throws an error when evaluated as a standard expression in an unindexed path.

### What's Wrong?

2. Steps to Reproduce (DDL & Indexes)
SQL
-- 1. Create Custom Tokenizer
CREATE INVERTED INDEX TOKENIZER IF NOT EXISTS ngram_2_tokenizer
PROPERTIES (
"type" = "ngram",
"min_gram" = "2",
"max_gram" = "2"
);

-- 2. Create Analyzer with lowercase filter
CREATE INVERTED INDEX ANALYZER ngram_2_analyzer_lc
PROPERTIES (
"tokenizer" = "ngram_2_tokenizer",
"token_filter" = "lowercase"
);

-- 3. Add Inverted Indexes to existing table
ALTER TABLE pdm_index_table
ADD INDEX idx_name_ngram2(name)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);

ALTER TABLE pdm_index_table
ADD INDEX idx_number_ngram2(number)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
ALTER TABLE xdm_xdmfilecontent
ADD INDEX idx_content_ngram2(attachment_content)
USING INVERTED
PROPERTIES (
"analyzer" = "ngram_2_analyzer_lc",
"support_phrase" = "true"
);
-- 4.build index
BUILD INDEX idx_name_ngram2 ON pdm_index_table;
BUILD INDEX idx_number_ngram2 ON pdm_index_table;
BUILD INDEX idx_content_ngram2 ON xdm_xdmfilecontent;
3. Comparison: What Works vs. What Fails
Case 1: Same-table OR (WORKS ✅)
● SQL
-- This query runs successfully using inverted index
-- success running 1
SELECT id FROM pdm_index_table
WHERE (name MATCH_PHRASE 'ghm' OR number MATCH_PHRASE 'ghm')
AND workingState <> 'CHECKED_OUT';
-- success running 2
select IndexA.id as 'IndexA_id'
from plm20_sit_xdm_f.pdm_index_table as IndexA
left join plm20_sit_xdm_f.pdm_index_master_table as IndexA$master on IndexA$master.id = IndexA._masterid
where (IndexA.className in ('Document'))
and ((IndexA.name match_phrase 'ghm' or
IndexA.number match_phrase 'ghm' )
and IndexA$master.isTemplate = 0)
and (IndexA.workingState <> 'CHECKED_OUT')
union all
select IndexA.id as 'IndexA_id'
from plm20_sit_xdm_f.pdm_index_table as IndexA
left join plm20_sit_xdm_f.pdm_index_master_table as IndexA$master on IndexA$master.id = IndexA._masterid
where (IndexA.className in ('Document'))
and ((
IndexA.id in (select SubSelect1FileContent.id as 'SubSelect1FileContent_id'
from plm20_sit_xdm_f.xdm_xdmfilecontent as SubSelect1FileContent
where SubSelect1FileContent.attachment_content match_phrase 'ghm' ))
and IndexA$master.isTemplate = 0)
and (IndexA.workingState <> 'CHECKED_OUT');

Case 2: OR with Subquery (FAILS ❌)
● SQL
-- This query triggers [NOT_IMPLEMENTED_ERROR]
SELECT IndexA.id
FROM pdm_index_table as IndexA
WHERE (
IndexA.number MATCH_PHRASE 'ghm'
OR IndexA.id IN (
SELECT SubContent.id
FROM xdm_xdmfilecontent as SubContent
WHERE SubContent.attachment_content MATCH_PHRASE 'ghm'
)
)
AND IndexA.workingState <> 'CHECKED_OUT';
4. Error MessageerrCode = 2, detailMessage = (192.168.10.235)[NOT_IMPLEMENTED_ERROR]Custom analyzer is not supported for unindexed MATCH operations.
5. Analysis For Case 1, Doris performs a Bitmap OR at the storage layer. However, for Case 2, the OR logic with a Semi-Join (Subquery) causes the match_phrase predicate to be hoisted to the VSelectNode (computational layer) instead of being pushed down to the VOlapScanNode. Since match_phrase with a custom analyzer requires an inverted index to function, it throws an error when evaluated as a standard expression in an unindexed path.

### What You Expected?

Expected Behavior
The query optimizer should be able to either:

Push down the match_phrase part to the inverted index separately.

Or handle the OR logic without breaking the indexed search path.

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.