influxdata / influxdata/influxdb

Getting unexpected values from diskBytes in the shard stats of the _internal database

Open
#24,478 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
1. `SELECT "diskBytes" FROM "shard" WHERE "database"='_internal'`
2. Note, I have started with a brand new database, and as such there is only a single /wal/ and /data/ directory path (so I know the problem is not due to [needing to sum across paths](https://community.influxdata.com/t/influxdb-usage-statistics/779/7)):
```
show tag values from "shard" with key = "path" where "database"='_internal'
name: shard
key value
--- -----
path /var/lib/influxdb/data/_internal/monitor/365
```

```
show tag values from "shard" with key = "walPath" where "database"='_internal'
name: shard
key value
--- -----
walPath /var/lib/influxdb/wal/_internal/monitor/365
```

__Expected behaviour:__
I expected the value returned by this metric to be a sum of the `/data/_internal` and `/wal/_internal` directories for this database. If this is not the case (due to some memory cacheing in the background?) then I would expect this value to be _at least greater than the sum_ of the `/data/_internal` and `/wal/_internal` directories. The reason I say this is because the [definition of diskBytes in the docs](https://docs.influxdata.com/platform/monitoring/influxdata-platform/tools/measurements-internal/#diskbytes) says "_including the size of the data directory and the WAL directory._"

__Actual behaviour:__
When examining the diskBytes field from shard (blue curve below), I find that it correctly tracks the size of the /wal/ + /data/ directories (orange curve, measused using `du`command) for this _internal database in the beginning, but then it drops. I found that these drops correspond to a new .wal file being created (and when this happens, the previous active .wal file reduces in size).

![image](https://github.com/influxdata/influxdb/assets/43373700/08b4e3d9-a39b-42b3-a7e1-902847fac170)

Regardless, this diskBytes does not seem to be a reliable indicator of the disk space used. Am I misunderstanding what diskBytes and shard represent, or are the docs misleading?

__Environment info:__

* InfluxDB v1.8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the SELECT against the shard measurement and comparing diskBytes with du for the _internal data and WAL paths, especially when a new .wal file is created. Read the linked diskBytes documentation and determine whether the metric or its documented meaning is wrong; done means the behavior and documentation agree and the reported discrepancy is covered by verification.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.