Icinga / Icinga/icinga2

Incorrect interpretation of units in perfdata normalization

Open
#9,486 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

  • #9609 by @Al2Klimov — closed without merging
area/checks area/itl enhancement TBD
Dominant language
C++
Stars
2.2k
Forks
616
Avg merge
5d 6h
Merged PRs (30d)
10

Description

Describe the bug

As documented, Icinga normalizes performance data before passing it to metric backends, so that MB, MiB and similar become bytes. MB is interpreted as base 10, so that if a plugin reports 33762MB, 33762000000 bytes are recorded.

The problem is that many plugins historically calculate MB and similar using base 2 (actually, I'm not aware of a plugin that would use base 10). For instance, running /usr/lib64/nagios/plugins/check_disk --help will output the following explanation:

Note: kB/MB/GB/TB are still calculated as their respective binary units due to backward compatibility issues.

The change of behaviour in Icinga was most probably introduced with the support for new UoMs, as noticed in #9060.

To Reproduce

  1. Configure disk monitoring using disk command (this is by default for the monitoring host).
  2. Run the relevant plugin /usr/lib64/nagios/plugins/check_disk -c 10% -w 20% -m manually to see the results.
  3. Run /usr/lib64/nagios/plugins/check_disk -c 10% -w 20% -m -u MiB and verify that the numbers are the same, though the units are shown as MiB now.
  4. Check the performance data recorded in the InfluxDB. It will be normalized by multiplying by 1000*1000.

Expected behavior

Binary MB should be normalized by multiplying by 1024*1024, so that 33762MB should become 35402022912 bytes.

Your Environment

  • Version used (icinga2 --version): r2.13.3-1
  • Operating System and version: Linux CentOS 7 (Core)
  • Enabled features (icinga2 feature list): api checker ido-pgsql influxdb mainlog notification
  • Version of check_disk (/usr/lib64/nagios/plugins/check_disk --version): v2.3.3 (nagios-plugins 2.3.3)

Possible workarounds

By default, check_disk is executed with -m option, so that the units are MB. It is possible to put vars.disk_units = "MiB" variable for disk command, so that the units become MiB, and get translated correctly.

However, not all plugins support such options. For example, Manubulon SNMP plugin check_snmp_storage.pl uses binary MB units (it can be verified by checking the source code), and there are no options to change that, so the fix is to modify the source code to output 'MiB' instead of 'MB'.

Another case is /usr/lib64/nagios/plugins/check_mem.pl which outputs KBs which actually mean KiBs.

I'm not sure if the right way is to change all the plugins to use the new unit system though. Maybe there should be an option in Icinga how to interpret MBs and similar - using base 10 or base 2.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Icinga's performance-data normalization and unit-of-measurement handling, then reproduce the discrepancy with check_disk using MB and MiB and compare the recorded InfluxDB values. Done means binary units are normalized consistently without changing the documented handling of explicit MiB units.

Written by the indexing model from the issue text.

Assessment

Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.