aws / aws/amazon-cloudwatch-agent

Delete not-update logs after a certain amount of time. [memory leak]

Open
#405 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

**Description:**
Currently, CloudWatch Agent only stop monitors logs file only if `auto_removal` was turned on and that applies to the case when logs files were not updated for a certain amount of time. However, this would leads to an issue of increasing files that Cloudwatch Agent needs to monitor, which would end up `run out of file descriptor` (meaning cannot open any more logs files to monitor) in most of the OS (including Linux). An example would be
```
2021-10-12T12:18:16Z E! [outputs.cloudwatchlogs] Aws error received when sending logs to ./audit: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: socket: too many open files
```

**Solution**:
Detect the newest changes when tailing and compare current time. If there was nothing changes regarding the log file, kill the tail process.

**Alternative solution:**
* [Increase max open file limit](http://woshub.com/too-many-open-files-error-linux/) on a particular OS
* Use [Log Sweeper](https://github.com/umbrellio/log_sweeper) to rotate and delete logs after a certain amount of time.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by tracing the log tailing path and how inactivity and auto_removal are currently handled; confirm how unchanged files accumulate monitored processes and file descriptors. Done means inactive log files stop being tailed after the defined period without affecting updated files, and the too-many-open-files failure is prevented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.