KDS CQEngine index is very slow for large OR queries

KDS CQEngine index is very slow for large OR queries

When querying on a single field with a many-claused OR equality filter (e.g. "name in ('amy', 'bob', etc)"). This may be caused by the deduplication strategy we pass to CQEngine (logical elimination). There are various hints that we may be able to leverage, e.g. if the filters are mutually exclusive.

A work-around is to disable indexing on the field.

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2396

Key: GEOMESA-2396
Type: Bug
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Mon, 17 Sep 2018 08:59:48 -0400
Updated: Tue, 14 Apr 2020 15:30:55 -0400
Affects Version(s): 2.0.2

Comments:

557058:48912df4-0004-4f90-9915-db26684ec006 (Tue, 14 Apr 2020 15:00:27 -0400):

Emilio Lahr-Vivaz any thoughts on this one?

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Tue, 14 Apr 2020 15:30:55 -0400):

I think this is still an issue.

Start by reading the KDS CQEngine index and query handling, then reproduce a large single-field OR equality query and compare indexed behavior with the workaround of disabling the field index. Done means the indexed query no longer shows the reported severe slowdown, with a regression check for large OR queries.