influxdata / influxdata/influxdb

Generic Event Markers populated by Influx queries

Open
#15,575 6 comments 2 reactions 0 assignees View on GitHub
kind/feature-request team/ui
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

(I believe @ebb-tide _may_ want to be aware of this submission specifically; hence, tagging her. By the way, thank you again, @ebb-tide, for your work on the CSV export functionality in Chronograf from way back. That was clutch for us.)

__Background:__
Originally, I submitted the core of this feature request as [Issue #5165](https://github.com/influxdata/chronograf/issues/5165), an enhancement to Chronograf's Annotation feature. My request got some love from others and was on the development roadmap for a short while. I assume that with the shift to Influx v2 it fell to the wayside. Most recently it was closed as stale.

I've heard from an insider the rough outline of the Event Marker feature in the Influx v2 roadmap. As I understand it, the first version to be rolled out will be tied specifically to Alerting & Monitoring, displaying notification events overlaid on data plots. Subsequently, Event Markers are intended to be made more generic and available throughout dashboards and data exploration.

I was encouraged to write up a feature request for sake of providing example use cases and providing some input on how the future generic Event Markers might operate.

__Proposal:__
Event Marker overlays throughout the InfluxDB v2 interface (e.g. Dashboards & Data Explorer) populated by Influx query results able to display both singular events and events with a time range. Event Markers would be centrally configured and available to be selectively enabled in the various views throughout the interface.

Manually entered events (akin to Chronograf's Annotation feature) would also be supported. However, it would probably make the best sense if manual annotations were stored in an Influx bucket and then queried for display using the same mechanisms needed to support the generic ability discussed here.

Notes on events:
* Many events are singular moments (e.g. a piece of equipment transitioned from low to high speed). These correspond well to individual rows in an Influx query response. The timestamp is the time of the event and its fields and tags contain its details.
* Some events are a time range (e.g. server downtime 7:39am – 8:02am or cleaning 1:37pm – 2:09pm). Mapping an event with a time range to Influx query results is a bit more challenging. A few options that might work:
* In a single query result row, the timestamp represents the beginning of the event while a field stores the end of the time range. This is only practical when the entirety of an event is known after its conclusion.
* Duration is stored as an integer field (cast with `toDuration()` in Flux).
* End time is stored as an integer field (cast with `toTime()` in Flux).
* End time is stored as a string field (parsed with `time()` in Flux).
* The event's time range is represented by two distinct rows in query results linked together logically by some unique ID in common. Care in display would be necessary in circumstances where the end of a time range is not yet stored. In the simplest case the beginning of the range could be displayed just like a singular event. Alternatively, some sort of “smear” of transparent color could visually indicate an event that has begun but has not concluded.

__Current behavior:__
To my knowledge Event Markers are in some form of development but only in relation to Alerting & Monitoring at present.

__Desired behavior:__
In essence, my proposal is essentially [Grafana's query-based Annotations](https://grafana.com/docs/reference/annotations/) plus the following. The first two in this list are enhancements to Grafana's existing abilities. The last item in this list is an all new feature.
1. Not only single event markers but also event time ranges (see _Proposal_ above) both populated by query results.
1. The ability to create centrally managed Event Marker queries whose results are available for overlay anywhere within Dashboards and the Data Explorer interface.
1. In Dashboards these would be selectable (toggle on / toggle off) for an entire dashboard but also per individual plot (dashboard cell).
1. In the Data Explorer interface not only would the user be able to enable preconfigured Event Marker overlays, they would also be able to author ad hoc Event Marker queries for overlay on the existing query submission / display interface.
1. Event Marker overlays on tabular data views. Some query results are not suitable for graphing, but understanding events in relation to query results is still desirable.

__Alternatives considered:__
At present we can use Grafana for query-filled “Annotations” in dashboards, albeit only for singular events and not time ranges. With some custom work Power BI can be coerced into displaying rudimentary event markers / annotations in dashboards and reports. Otherwise, we have not found a good way to present discrete events or event time ranges in a user-driven data explorer interface for InfluxDB time-series data. Chronograf was previously our best hope for this had my requested enhancement to Annotations come to pass.

__Use case:__
Our time-series data largely falls in three categories:
1. Sensor readings (environmental monitoring)
1. Equipment operation (on/off, cycle times, state transitions)
1. Production line events — e.g. Batch XYZ entered/exited a production stage

All of the above are interrelated and also influenced by other events — maintenance, system changes, etc. We want to begin tracking external events also as time-series data (a fourth category that does not yet exist for us) so we can visually correlate cause and effect across our systems.

Examples:
* A particular environmental reading may change as the result of equipment operation. A reading may rise or fall in step with a particular piece of equipment turning on or off.
* Equipment operation can be influenced by maintenance or lack thereof. A machine's cycle time may be correlated with when its filters were last cleaned.
* Production line events may shift in relation to environmental conditions or equipment downtime. Such shifts correlated to other events can reveal unreported operations issues.

Being able to visualize our events in relation to plotted data such as above would help us:
* Troubleshoot quality, environmental, and performance issues
* Provide greater situational awareness during regular operations on the shop floor
* Provide valuable input to various R&D & continuous improvements efforts

Contributor guide

Open the contributing guide

Research direction

The issue does not name implementation files or tests. Start by reviewing the existing Alerting & Monitoring Event Marker work and the Dashboard/Data Explorer paths; define query-backed singular and ranged markers, central configuration, and overlays as the completion criteria.

Written by the indexing model from the issue text.

Assessment

Domain
data-visualization, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.