aws / aws/amazon-cloudwatch-agent

v1.300070.0 leaks "dbus-daemon --session" procs with Prometheus/AppInsights SAP config, exhausts inotify ("too many open files") - fine on v1.300069.0

Open
#2,195 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

**Describe the bug**

We run the agent across a fleet of EC2 Linux hosts and started getting "too many open files" out of the agent, plus a pile of `dbus-daemon: Cannot initialize inotify` in the journal. It traces back to leaked dbus-daemon processes.

On hosts where CloudWatch Application Insights (SAP monitoring) has added a prometheus scrape to the agent config, v1.300070.0 spawns a session bus (`dbus-daemon ... --session`, looks like a glib autolaunch) every time the agent starts or the config gets re-applied. The systemd unit ships with `KillMode=process`, so when the service stops those dbus procs don't get killed - they stay in the service cgroup and pile up. Each one holds an inotify instance + fds, so after a few days they blow past `fs.inotify.max_user_instances` (128 for root) and the agent can't add file watchers anymore.

v1.300069.0 doesn't do this. Same host, same config, same uptime, zero session dbus procs. It only started after the box auto-updated to v1.300070.0 (AWS-ConfigureAWSPackage pulling latest) at the end of June.

**Steps to reproduce**

- EC2 Linux, systemd + cgroup v2, agent running as root, installed via `AWS-ConfigureAWSPackage` (latest = v1.300070.0)
- Enable Application Insights SAP monitoring so it drops a prometheus config into `amazon-cloudwatch-agent.d/` (file_sd scrape of a `SAP_HOST_EXPORTER_*` job)
- Let the config get re-applied periodically (State Manager etc) so the agent restarts
- `watch 'pgrep -x dbus-daemon | wc -l'` - the count climbs and never comes back down, and the journal fills with `(dbus-daemon) remains running after unit stopped`

**What did you expect to see?**

The agent shouldn't be autolaunching a session bus for prometheus/host collection, and whatever it does spawn should get cleaned up when the service stops instead of leaking.

**What did you see instead?**

dbus procs accumulating without limit. Log lines (hostnames/pids scrubbed):

```
systemd[1]: amazon-cloudwatch-agent.service: Unit process NNNN (dbus-daemon) remains running after unit stopped.
dbus-daemon[NNNN]: Cannot initialize inotify
start-amazon-cloudwatch-agent[NNNN]: level=ERROR source=file.go:243 msg="Error adding file watcher" discovery=file config=SAP_HOST_EXPORTER_xxxx err="too many open files"
```

The leaked procs look like:
```
dbus-daemon --syslog --fork --print-pid N --print-address N --session
```

For scale: on one box we went from 0 "remains running" events/week on v1.300069.0 to ~50k/day the week after it moved to v1.300070.0, nothing else changed. They're all in the `amazon-cloudwatch-agent.service` cgroup, which is why `KillMode=process` lets them survive the stop.

**What version did you use?**

v1.300070.0 (b1586, built with go1.26.4) is broken. v1.300069.0 (go1.26.3) is fine. Only 070 reproduces it for us.

**What config did you use?**

The prometheus config Application Insights generates (ids redacted):
```json
{"logs":{"metrics_collected":{"prometheus":{
"cluster_name":"applicationinsights-prometheus-cluster",
"log_group_name":"Applicationinsights-Prometheus",
"prometheus_config_path":"/opt/aws/amazon-cloudwatch-agent/applicationinsights_prometheus.yaml",
"emf_processor":{"metric_declaration":[{"source_labels":["job"],
"label_matcher":"^SAP_HOST_EXPORTER_xxxx$",
"metric_selectors":["^sap_alerts_Availability$"]}]}}}}}
```
agent block uses `run_as_user: root`.

**Environment**

OS: Oracle Linux 9.7, kernel 5.15 UEK, cgroup v2, systemd 252. Stock `amazon-cloudwatch-agent.service` unit (`KillMode=process`, `Restart=on-failure`, `RestartSec=60s`).

**Additional context**

A couple of things I'm hoping you can help with:

- Any idea what changed between 069 and 070 on the prometheus/host path (or a bumped dependency) that makes it autolaunch a session bus now? That looks like the real trigger.
- Separate from that, could the packaged unit move to `KillMode=mixed`? That reaps the leftover dbus on stop and keeps this from snowballing regardless of what spawns it. The `KillMode=process` unit also comes up in #1320.

For anyone else who hits this: removing the AppInsights prometheus config stops new leaks, `pkill -f 'dbus-daemon .*--session'` reclaims the fds, and pinning the package to v1.300069.0 avoids it.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the Application Insights Prometheus config at /opt/aws/amazon-cloudwatch-agent/applicationinsights_prometheus.yaml and compare the Prometheus/host collection behavior between v1.300069.0 and v1.300070.0. Inspect the packaged amazon-cloudwatch-agent.service, especially KillMode=process. Done means the agent no longer autolaunches or leaks session dbus-daemon processes and stopped services do not leave them behind.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux, prometheus
Domain
devops, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.