influxdata / influxdata/influxdb
Today function not accounting for timezone
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
**Steps to reproduce**
1. My test data is the hour number written at every 15th and 45th minute of the hour. The midnight hour is written as 24 instead of 0 for purpose of this test. Data is written in UTC, and I am in Europe/Amsterdam UTC+1, so data looks like this:
```
_start | _stop | _time | _value | _field | _measurement | host | topic
-- | -- | -- | -- | -- | -- | -- | --
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-15T23:15:00.091801843Z | 24 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-15T23:45:00.119755916Z | 24 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T00:15:00.064616933Z | 1 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T00:45:00.14960772Z | 1 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T01:15:00.088386139Z | 2 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T01:45:00.1210529Z | 2 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T02:15:00.120252814Z | 3 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T02:45:00.070592294Z | 3 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T03:15:00.118206701Z | 4 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T03:45:00.079333413Z | 4 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T04:15:00.079558896Z | 5 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T04:45:00.086431761Z | 5 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T05:15:00.153160125Z | 6 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T05:45:00.075220488Z | 6 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T06:15:00.081396106Z | 7 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T06:45:00.078725626Z | 7 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T07:15:00.103096574Z | 8 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T07:45:00.1413385Z | 8 | hour | mqtt_consumer | telegraf_weather | influxtest
2022-02-15T08:15:14.770899515Z | 2022-02-16T08:15:14.770899515Z | 2022-02-16T08:15:00.106263549Z | 9 | hour | mqtt_consumer | telegraf_weather | influxtest
```
It is currently 9:16 AM local time, so 8:16Z.
2. Use the following query to get the maximum value:
```
import "timezone"
option location = timezone.location(name: "Europe/Amsterdam")
from(bucket: "influxtest")
|> range(start: -1d)
|> filter(fn: (r) => r._field == "hour")
|> filter(fn: (r) => r.topic == "influxtest")
|> aggregateWindow(every: 1d, fn: max, createEmpty: false)
```
Outcome:
```
_start | _stop | _time | _value | _field
-- | -- | -- | -- | --
2022-02-15T08:17:13.517420545Z | 2022-02-16T08:17:13.517420545Z | 2022-02-15T23:00:00Z | 23 | hour
2022-02-15T08:17:13.517420545Z | 2022-02-16T08:17:13.517420545Z | 2022-02-16T08:17:13.517420545Z | 24 | hour
```
_Outcome is correct_ (Max value for the 15th is indeed 23 but is not relevant to this test)
3. Now, get the max value for "today so far" only, using today() function:
```
import "timezone"
option location = timezone.location(name: "Europe/Amsterdam")
from(bucket: "influxtest")
|> range(start: today())
|> filter(fn: (r) => r._field == "hour")
|> filter(fn: (r) => r.topic == "influxtest")
|> aggregateWindow(every: 1d, fn: max, createEmpty: false)
```
Outcome:
```
_start | _stop | _time | _value | _field
-- | -- | -- | -- | --
2022-02-16T00:00:00Z | 2022-02-16T08:19:00.96465188Z | 2022-02-16T08:19:00.96465188Z | 9 | hour
```
_Outcome is incorrect_. Should be 24
**Expected behaviour**
If option location is specified, today() should calculate aggregates starting at local midnight
**Actual behaviour**
today() starts at UTC midnight disregarding the location option. This influences all aggregate functions. This test was also done with the mean aggregate, which also produces the wrong result when using the today() function
**System and version**
InfluxDB runs as a Docker container on a Docker Swarm cluster on Raspberry Pi, with the following specific settings:
weather:
image: influxdb:2.1.1-alpine
environment:
- TZ=Europe/Amsterdam
volumes:
- /docker/containers/influxdb/weather/etc:/etc/influxdb2
- /docker/containers/influxdb/weather/data:/var/lib/influxdb2
- /etc/localtime:/etc/localtime:ro
Pi's are running the latest 64-bit official OS
**Additional info**
I created tasks to calculate aggregates and put them into separate buckets. For daily aggregates for example, this would probably work if the task would run daily at exactly midnight only. However, I wanted the tasks to be restartable, so I can run them manually to fix a possible erroneous situation. They then should yield the same result. Also, I like to run the aggregate tasks more frequently than the aggregate period. So run the daily aggregate task every hour, so I can see the aggregate value for "today so far". Re-running the task and running it more frequently will cause an extra data point to be written, while it should be a maximum of one per day. Using the timeSrc: "_start" and today() functions, the _time field for each day is always the same. This should solvemy issues and make the task restartable and able to run more frequently.
For example, my weekly aggregate task would have the following code:
```
import "date"
import "timezone"
import "experimental"
option task = {name: "agg_weekly", every: 1d, offset: 5m}
option location = timezone.location(name: "Europe/Amsterdam")
// Duration from first weekday
offsetToFirstDayOfWeek = "-" + string(v: date.weekDay(t: now())) + "d"
// today() returns midnight, which is the starting point.
// Subtract offsetToFirstDayOfWeek to make the starting point midnight
// on the first day of the week
offset = experimental.subDuration(
d: duration(v: offsetToFirstDayOfWeek),
from: today(),
)
```
The data range start will be 'offset', and the aggregateWindow "every" will be '1w'. However, the average value of the "current week so far" will be calculated daily.
Because today() starts at UTC midnight however, it will yield incorrect results for each day.
Contributor guide
Research direction
Start by tracing today() with option location = timezone.location(name: "Europe/Amsterdam"), then compare its range start with aggregateWindow(every: 1d) across the supplied examples. Done means today() begins at local midnight when a location is configured, so daily and weekly aggregates include the expected local-day data.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100