opensearch-project / opensearch-project/alerting

[FEATURE] Monitor explain API

Open
#751 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Kotlin
Stars
82
Forks
133
Avg merge
2d 11h
Merged PRs (30d)
9

Description

Is your feature request related to a problem?
Just like in ISM, user could benefit from an explain API for monitors, i.e. for document level monitors, API could be used to track monitor's current progress - number of currently accumulated "work" for a monitor to be done on the next monitor run (in the moment of explain invocation of course as this number could change until the run itself), or put differently, an information about how behind on its work the monitor currently is - accumulated work since last run. This information could be displayed in few ways depending on context.

What solution would you like?
Implementation of the explain API on top off monitor endpoint where requests containing valid monitor id would result in a response with cumulative count of shard updates that happened since previous scheduled run of the monitor (cumulative sequence number differences for monitor's index/indices shards).
For a certain scenarios, an optional request parameter called i.e. count_docs could be passed if exact document count (currently accumulated number of documents for checking since the last monitor run) is to be retrieved instead. This may be handy as index updates without document insertion would also generate seq_no updates and may provide ambiguous information.

REQUEST

GET /_plugins/_alerting/monitors/<monitorID>/_explain?count_docs=<true/false>

Request url params:

Param Description
monitorID ID of the document-level monitor
count_docs Include document difference in response - false if not specified

RESPONSE

{
    "_id" : <monitorID>,
    "documents_behind" : <documents_behind>,
    "last_run_timestamp" : <last_run_timestamp>,
    "query_index_list" : <query_index_list>
}

Response body params:

Param Description
_id ID of the targeted monitor
documents_behind Accumulated documents/seq_no difference depending on count_docs
last_run_timestamp Timestamp of monitor's last run
query_index_list List of monitor's query indexes

We can go one step further by utilizing the fact that the documents are being counted on every run. By storing these values for last N monitor runs we could track and return the trend of work that monitor has recently gone through and associate it with potential monitor "lateness" or "delay".

Do you have any additional context?
The count_docs version of the previously mentioned request would come in play when index is prone to both document updates and insertions. With both doc and seq_no counting options user would be able to distinguish between updates and insertions when needed.
When dealing with i.e. logs, invoking the API only for seq_no difference would be sufficient as we are confident the documents would always be appended and never updated. Also, counting the documents comes at potential performance cost as the whole index search would be required, so it is to be omitted when not of interest.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the monitor endpoint, GET /_plugins/_alerting/monitors//_explain, and trace how document-level monitor runs record accumulated work and the last run timestamp. Define how sequence-number and optional document counts are obtained, then verify the response fields and count_docs behavior for valid monitor IDs, including the stated performance trade-off.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.