influxdata / influxdata/telegraf

inputs.snmp does not reconnect after network issue

Open
#16,880 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[[inputs.snmp]]
alias = "CENSORED"
name_override = "CENSORED"
agents = ["TARGET.com,..."]
timeout = "5s"
version = 3
sec_name = "CENSORED"
auth_protocol = "SHA"
auth_password = "CENSORED"
sec_level = "authPriv"
priv_protocol = "AES"
priv_password = "CENSORED"
agent_host_tag = "source"

[[inputs.snmp.field]]
oid = ".1.3.6.1.2.1.1.3.0"
name = "uptime"
```

### Logs from Telegraf

```text
Apr 25 09:37:20 HOSTNAME.com telegraf[1405]: 2025-04-25T09:37:20Z E! [inputs.snmp::CENSORED] Error in plugin: agent udp://TARGET.com:161: performing get on field uptime: request timeout (after 3 retries)
Apr 25 09:36:20 HOSTNAME.com telegraf[1405]: 2025-04-25T09:36:20Z E! [inputs.snmp::CENSORED] Error in plugin: agent udp://TARGET.com:161: gathering table hrProcessorTable: performing bulk walk for field percpu_load_percent: request timeout (after 3 retries)
...
ABOVE ERRORS REPEATS FOREVER
...
RESTART TELEGRAF
...
NO MORE ERRORS
```

### System info

Telegraf 1.34.1 (git: HEAD@6d52cd2f), RHEL9 latest updates

### Docker

_No response_

### Steps to reproduce

1. Add firewall (but likely can be any other network change or temporary issue) in front of target system being monitored through telegraf inputs.snmp
2. Telegraf stops collecting data
3. Restarting telegraf resolves the issue
...

### Expected behavior

Telegraf reconnects after error is detected

### Actual behavior

Telegraf errors out forever

### Additional info

Hi, we have added a firewall in front of a system being monitored by inputs.snmp, after that the already mentioned error appears in logs indefinitely (in our case 24 hours+). Restarting telegraf resolved the issue.

There seems to be some sort of connection cache, it even has a Reconnect implemented on error, but for some reason it doesnt kick in / works correctly in this case.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the inputs.snmp plugin's connection cache and reconnect-on-error handling, then reproduce the reported firewall or network interruption using the supplied Telegraf configuration and logs. Done means SNMP collection resumes after the network recovers without restarting Telegraf, with coverage for the failure and recovery path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.