opensearch-project / opensearch-project/security-analytics
[FEATURE] Warning system when detector query becomes too large
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 111
- Forks
- 111
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 13
Description
Is your feature request related to a problem?
A detector can potentially be configured with thousands of rules. Rules are used to construct the query that's executed by the detector to generating finds/alerts. If a query becomes too large, the execution of the query could fail with ParseException. E.g.,
ParseException[Query string length exceeds max allowed length 32000 (search.query.max_query_string_length); actual length: 32920];
Increasing the indices.query.bool.max_clause_count and search.query.max_query_string_length cluster settings can resolve those errors, but this is not a good long-term solution as large queries can negatively impact cluster performance (e.g., causing out-of-memory errors by consuming large amounts of heap memory, increasing search latency by taking a long time to parse and complete large queries, etc.). This would likely require scaling up/out the cluster to compensate for the performance issues. This is a limitation of opensearch itself, not so much the plugin.
What solution would you like?
It would be helpful for the plugin to intelligently warn when a detector query is becoming too large so that users can take appropriate action (e.g., adjust the 2 cluster settings mentioned above, auditing rules to make them more succinct/efficient, create more detectors with fewer rules, etc.).
This could be accomplished through the plugin UI as part of a dry-run process when creating/updating a detector.
It would also be helpful for the detectors to generate error alerts that call out the parse exception.
Do you have any additional context?
Ref# V2112755895
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how detector rules are assembled into queries and how detector creation or updates could support a dry run. Then inspect the handling of ParseException and detector error alerts; done should include a useful warning for oversized queries and an alert that identifies the parse failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- full-stack
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100