influxdata / influxdata/influxdb
Getting unexpected values from diskBytes in the shard stats of the _internal database
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
1. `SELECT "diskBytes" FROM "shard" WHERE "database"='_internal'`
2. Note, I have started with a brand new database, and as such there is only a single /wal/ and /data/ directory path (so I know the problem is not due to [needing to sum across paths](https://community.influxdata.com/t/influxdb-usage-statistics/779/7)):
```
show tag values from "shard" with key = "path" where "database"='_internal'
name: shard
key value
--- -----
path /var/lib/influxdb/data/_internal/monitor/365
```
```
show tag values from "shard" with key = "walPath" where "database"='_internal'
name: shard
key value
--- -----
walPath /var/lib/influxdb/wal/_internal/monitor/365
```
__Expected behaviour:__
I expected the value returned by this metric to be a sum of the `/data/_internal` and `/wal/_internal` directories for this database. If this is not the case (due to some memory cacheing in the background?) then I would expect this value to be _at least greater than the sum_ of the `/data/_internal` and `/wal/_internal` directories. The reason I say this is because the [definition of diskBytes in the docs](https://docs.influxdata.com/platform/monitoring/influxdata-platform/tools/measurements-internal/#diskbytes) says "_including the size of the data directory and the WAL directory._"
__Actual behaviour:__
When examining the diskBytes field from shard (blue curve below), I find that it correctly tracks the size of the /wal/ + /data/ directories (orange curve, measused using `du`command) for this _internal database in the beginning, but then it drops. I found that these drops correspond to a new .wal file being created (and when this happens, the previous active .wal file reduces in size).

Regardless, this diskBytes does not seem to be a reliable indicator of the disk space used. Am I misunderstanding what diskBytes and shard represent, or are the docs misleading?
__Environment info:__
* InfluxDB v1.8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the SELECT against the shard measurement and comparing diskBytes with du for the _internal data and WAL paths, especially when a new .wal file is created. Read the linked diskBytes documentation and determine whether the metric or its documented meaning is wrong; done means the behavior and documentation agree and the reported discrepancy is covered by verification.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100