influxdata / influxdata/influxdb

Support relative / delta timestamps (i.e. duration since collection) so that devices don't have to keep an accurate clock time

Open
#23,342 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Proposal:__
I would like to see support for relative/delta timestamps, i.e. reporting how long ago a datapoint was instead of its absolute epoch timestamp. This would allow devices to send historical data while alleviating the need for accurate timekeeping on those devices.

__Current behavior:__
Timestamps can currently be attached to reported data in an epoch format, i.e. how many nanoseconds (or some other unit) since the Unix epoch.

__Desired behavior:__
A negative timestamp value sent with a datapoint would be treated as "x amount of time ago", in where x is some amount of time in whatever precision is configured. This duration would be subtracted from the server's UTC timestamp.

For example, a line protocol message might look like this:

```
measurementName,tagKey=tagValue fieldKey="fieldValue" -13200000000
```

Where `-13200000000` represents 13.2 seconds ago (in nanoseconds).

If the server's current time was `1556813561098000000`, it would subtract `13200000000` from it and write the event with the timestamp `1556813547898000000`.

The use of a negative number is just an example; some other delta indicator (e.g. `T-13200000000`) could be used instead.

__Alternatives considered:__
The primary alternative is for all devices involved to synchronise themselves to absolute clock time using NTP. This is undesirable for a number of reasons:

1. increased complexity, particularly in embedded devices;
2. additional points of failure;
3. if internet access is not available, a local NTP server must be deployed and maintained.

I am unaware of any other alternative approach that does not require additional complexity and infrastructure.

__Use case:__
Embedded devices are often independently capable of keeping track of _elapsed_ time reliably and accurately, but require additional resources and effort (e.g. NTP sync) to keep an accurate and synchronised _clock_ time. Normally a device can defer timekeeping to the server by omitting the timestamp, but this prevents the device from reporting historical data.

The ability to report historical data is particularly desirable in the event of a network outage, since it allows the device to buffer data and batch report it once the outage is resolved.

If influxdb supported delta timestamps, devices could report historical data without needing to independently keep track of clock time. This would be the best of both worlds: devices can delegate timekeeping to the server, but also keep the ability to report historical data.

Contributor guide

Open the contributing guide

Research direction

Start by tracing InfluxDB's line protocol timestamp parsing and the server UTC timestamp handling. Done means a datapoint with a relative or delta timestamp can be written as historical data while existing absolute timestamps continue to work; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.