opensearch-project / opensearch-project/security-analytics

[FEATURE] Warning system when detector query becomes too large

Open
#1,655 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
111
Forks
111
Avg merge
1d 17h
Merged PRs (30d)
13

Description

Is your feature request related to a problem?
A detector can potentially be configured with thousands of rules. Rules are used to construct the query that's executed by the detector to generating finds/alerts. If a query becomes too large, the execution of the query could fail with ParseException. E.g.,

ParseException[Query string length exceeds max allowed length 32000 (search.query.max_query_string_length); actual length: 32920];

Increasing the indices.query.bool.max_clause_count and search.query.max_query_string_length cluster settings can resolve those errors, but this is not a good long-term solution as large queries can negatively impact cluster performance (e.g., causing out-of-memory errors by consuming large amounts of heap memory, increasing search latency by taking a long time to parse and complete large queries, etc.). This would likely require scaling up/out the cluster to compensate for the performance issues. This is a limitation of opensearch itself, not so much the plugin.

What solution would you like?
It would be helpful for the plugin to intelligently warn when a detector query is becoming too large so that users can take appropriate action (e.g., adjust the 2 cluster settings mentioned above, auditing rules to make them more succinct/efficient, create more detectors with fewer rules, etc.).

This could be accomplished through the plugin UI as part of a dry-run process when creating/updating a detector.

It would also be helpful for the detectors to generate error alerts that call out the parse exception.

Do you have any additional context?
Ref# V2112755895

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how detector rules are assembled into queries and how detector creation or updates could support a dry run. Then inspect the handling of ParseException and detector error alerts; done should include a useful warning for oversized queries and an alert that identifies the parse failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
full-stack
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.