prometheus / prometheus/node_exporter

Index out of range error in parseMemInfo for linux

Open
#2,751 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

Host operating system: output of uname -a

Linux version 5.4.0-153-generic (buildd@bos03-amd64-008) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023

node_exporter version: output of node_exporter --version

node_exporter, version 1.6.0

node_exporter command line flags

/bin/sh -c /usr/local/bin/node_exporter --web.listen-address=:42000 --collector.systemd --collector.processes

node_exporter log output
panic: runtime error: index out of range [1] with length 1

goroutine 218975 [running]:
github.com/prometheus/node_exporter/collector.parseMemInfo({0xbfb740, 0xc000095460})
  /app/collector/meminfo_linux.go:56 +0x2b9
github.com/prometheus/node_exporter/collector.(*meminfoCollector).getMemInfo(0x0)
  /app/collector/meminfo_linux.go:40 +0xf0
github.com/prometheus/node_exporter/collector.(*meminfoCollector).Update(0xc000215150, 0xc000319d40)
  /app/collector/meminfo.go:50 +0x3e
github.com/prometheus/node_exporter/collector.execute({0xb1390b, 0xc00041a1e0}, {0xbfa9a0, 0xc000215150}, 0xc00041a240, {0xbfa3c0, 0xc000031c40})
  /app/collector/collector.go:161 +0x9c
github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xb1390b, 0xc00030e790}, {0xbfa9a0, 0xc000215150})
  /app/collector/collector.go:152 +0x3d
created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
  /app/collector/collector.go:151 +0xd5

Are you running node_exporter in Docker?

no

What did you do that produced an error?

My setup is several lxc containers with node exporters. One day after host machine rebooting node exporters started to "crash" with these errors in logs from time to time.

What did you expect to see?

I found a similar "issue" - https://github.com/prometheus/node_exporter/pull/1671 . It says it happened because of empy lines in /proc/meminfo. Perhaps in my case instead of empty line I get incorrect line without key or value and "parts" length in code of parseMemInfo is 1 what caused the error bacuse there is parts[1] in the code.

I dont really understand why it happened in my system, i couldn't catch "incorrect lines" in /proc/meminfo, maybe because it happening randomly.

Perhaps it should be fixed with additional validation like in https://github.com/prometheus/node_exporter/pull/1671 but for parts length equal to 1. Or perhaps you just could give me some advise what is a better way to deal with my problem because it seems like it is maybe more my local problem with OS and not problem with exporters.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in collector/meminfo_linux.go at parseMemInfo, especially the reported line 56, and compare its handling with the validation added in pull request #1671. Reproduce the malformed or incomplete /proc/meminfo input described in the issue, then verify that parsing no longer panics while normal meminfo data still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
observability, operating-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.