influxdata / influxdata/influxdb
Feature Request: Provide a metric in the _internal database which tracks the total accurate disk usage of each database
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Use case:__
Obtaining total database sizes on disk for monitoring purposes.
__Proposal:__
It would be great to have a single metric in the [_internal database](https://docs.influxdata.com/platform/monitoring/influxdata-platform/tools/measurements-internal/) which indicates the accurate values of the total disk space used by each database. This would ideally include the sum of the /data/"database_01" and /wal/"database_01" directories, and have the database name appear as a tag.
__Current behaviour:__
Currently there is a metric called ["**diskBytes**"](https://docs.influxdata.com/platform/monitoring/influxdata-platform/tools/measurements-internal/#diskbytes) in the "**shard**" measurement of the _internal database. However, this has two problems:
1. [It does not return](https://github.com/influxdata/influxdb/issues/24478) accurate values of the /data/ and /wal/ directories, but rather depends on some purging which is going on in the background (despite what is stated [in the docs](https://docs.influxdata.com/platform/monitoring/influxdata-platform/tools/measurements-internal/#diskbytes)).
2. The other issue with the current diskBytes metric is that it returns results which are tagged according to the tag key "path":
```
show tag values from "shard" with key = "path" where "database"='_internal'
name: shard
key value
--- -----
path /var/lib/influxdb/data/_internal/monitor/365
path /var/lib/influxdb/data/_internal/monitor/366
path /var/lib/influxdb/data/_internal/monitor/368
```
This means that in order to retrieve the total that I am looking for, you need to sum the values over all paths, like this:
```
SELECT sum(diskBytes) FROM "shard" WHERE "database"='_internal' AND time>now()-2m GROUP BY time(5s)
name: shard
time sum
---- ---
1700825990000000000
1700825995000000000
1700826000000000000 66281405
1700826005000000000
1700826010000000000 66820082
1700826015000000000
1700826020000000000 67359027
1700826025000000000
1700826030000000000 67897018
1700826035000000000
1700826040000000000 68435074
1700826045000000000
1700826050000000000 68975017
1700826055000000000
1700826060000000000 58733801
1700826065000000000
1700826070000000000 59272647
1700826075000000000
1700826080000000000 59811046
1700826085000000000
1700826090000000000 60350409
1700826095000000000
1700826100000000000 60889485
1700826105000000000
1700826110000000000 61428518
```
The problem is that if I want to view a downsampled version of this (say 1 value every 30 sec), then it requires a subquery:
```
SELECT first(diskBytes_sum) FROM (
SELECT sum(diskBytes) AS diskBytes_sum FROM "shard"
WHERE "database"='_internal' AND time>now()-2m GROUP BY time(5s)
) GROUP BY time(1m)
name: shard
time first
---- -----
1700826000000000000 66281405
1700826060000000000 58733801
```
So, even though I require data only every 1minute, I have to first query all the underlying data (every 5s) in order to accurately do the sum over all paths. This is very inefficient.
__Alternatives considered:__
Currently, in order to get the disk space of the /wal/ and /data/ directories for each database, I have to run telegraf with the [exec input plugin](https://github.com/influxdata/telegraf/blob/master/plugins/inputs/exec/README.md), which executes a shell script that runs `du` on the directories. This presents other problems, for example it's fiddly to get all the permissions correct on the script, so that the containers are able to access InfluxDB's data and wal directories to read them. It works okay, but would be much simpler if InfluxDB could expose such values, since it already should have access to them.
Contributor guide
Research direction
Start by reviewing the _internal database's shard measurement and its existing diskBytes metric, along with the linked diskBytes documentation and issue. Define the implementation scope for measuring both data and WAL usage and aggregating it by database; done means an accurate metric with the database name as a tag.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, shell
- Domain
- databases, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100