prometheus / prometheus/node_exporter

`NF_CONNTRACK_PROCFS` is obsolete and disabled by default in recent kernels

Open
#2,491 5 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

accepted enhancement help wanted platform/Linux
Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

Host operating system: output of uname -a

Linux carsten 5.15.70 #1-NixOS SMP Fri Sep 23 12:15:52 UTC 2022 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version
node_exporter, version 1.3.1 (branch: unknown, revision: v1.3.1)
  build user:       nix@nixpkgs
  build date:       unknown
  go version:       go1.17.13
  platform:         linux/amd64

(Though the problem also appears to exist on master).

node_exporter command line flags
ExecStart=/nix/store/xvib5wqz29v3m3ln0266w7w3ncdkfwgg-node_exporter-1.3.1/bin/node_exporter \
  --collector.textfile \
   \
  --web.listen-address 0.0.0.0:9100 --collector.textfile.directory=/run/prometheus-node-exporter-textfiles
Are you running node_exporter in Docker?

no

What did you do that produced an error?

I updated my host's Linux kernel to a version >5.15.65. In this version, the default value of NF_CONNTRACK_PROCFS was changed to n. This means that /proc/net/stat/nf_conntrack is not available anymore and thus node_scrape_collector_success{collector="conntrack"} is 0 (and all metrics except for nf_conntrack_entries and nf_conntrack_entries_limit which are collected via /sys/net/netfilter/nf_conntrack_{count,max} are not available anymore.

The commit message suggests to use conntrack(8) instead of /proc/net/stat/nf_conntrack because the latter one was marked as obsolete.

What did you expect to see?

I'd expect to see all metrics of the conntrack collector being exposed.

What did you see instead?

Only _limit and _entries were exposed instead.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the conntrack collector and its use of /proc/net/stat/nf_conntrack, then investigate the conntrack(8) approach suggested in the issue for kernels newer than 5.15.65. Done means the collector succeeds and exposes the metrics that are currently missing, while retaining the entries and limit metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
observability-sre, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.