influxdata / influxdata/influxdb

Plans for a "Counter" Data Type: a data type which increment its value each time based on the last point

Open
#23,920 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Proposal:__

In prometheus, there is a `counter` data type: see [here](https://prometheus.io/docs/concepts/metric_types/#counter).
The purpose of this is to write easily a new Point based on a previous Point values.
This is especially useful when you want the number of requests served, tasks completed, errors, messages sent, event received?etc: to represent any cumulative values.

I would hope to see a similar data structure in Influx DB.

__Current behavior:__

Our current implementation is the (1) choice exposed in "alternative considered".

With the help of the cache (oldPoint), we then write our new Point like that:
```js
new Point("METRIC")
.tag("group", groupID)
.tag("user", userID)
.stringField("type", type)
.intField("count", count)
.intField("value1", oldPoint.value1 + 1)
.intField("value2", oldPoint.value2 + 10)
.intField("value3", oldPoint.value3 + 5)
.intField("value4", oldPoint.value4 + 2);
```

For the old Point `value1` was 50, the new Point `value1` is 51.

__Desired behavior:__

With the `Counter` data type, we no longer need cache.

Based on the tags, InfluxDB find the last Point with the exact same tags and then add a new Point. For this new Point, all `incrementField` get incremented based on the last Point value for the incrementField of the same name.

The API could look like this:
```js
new Counter("METRIC")
.tag("group", groupID)
.tag("user", userID)
.stringField("type", type)
.intField("count", count)
.incrementField("value1", 1)
.incrementField("value2", 10)
.incrementField("value3", 5)
.incrementField("value4", 2);
```

The last Point `value1` was 50, the new Point `value1` is 51.

__Alternatives considered:__

Right now, I have two choices:

- (1) cache the previous value and increment it when creating a new point
- (2) adding the points without the cumulative, and use `cumulative_sum` when reading

These 2 approaches have major drawbacks:
- (1) It relies heavily on the cache and its stability. It's a SPOF, and the issue is even more important with replication and concurrent access to the Database.
- (2) It adds computing time when reading data. I also need to do a `cumulative_sum` from the beginning of time even if I want the data for the last day only. Also as said in "desired behaviour", this solution makes it harder to handle reading data with tags in more complex queries.

__Use case:__

This feature already exists in Prometheus. This makes us wonder about switching if it becomes a necessity. I am sure others are in the same case as we are.
Besides any cumulative values (number of events, messages, requests etc..), it is also useful for global stats overall (based on tags).
For exemple for temperatures, we could use this to track how often each thermometers recorded temperature.

Contributor guide

Open the contributing guide

Research direction

The issue names no implementation files, tests, or entry points. Start by locating InfluxDB's point-writing and field-type handling, then review the proposed Counter semantics and existing cumulative-sum behavior; done would require an agreed design, implementation scope, and tests for tag matching, increments, and concurrent writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, prometheus
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.