aws / aws/amazon-cloudwatch-agent
Delete not-update logs after a certain amount of time. [memory leak]
- Dominant language
- Go
- Stars
- 550
- Forks
- 271
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 13
Description
**Description:**
Currently, CloudWatch Agent only stop monitors logs file only if `auto_removal` was turned on and that applies to the case when logs files were not updated for a certain amount of time. However, this would leads to an issue of increasing files that Cloudwatch Agent needs to monitor, which would end up `run out of file descriptor` (meaning cannot open any more logs files to monitor) in most of the OS (including Linux). An example would be
```
2021-10-12T12:18:16Z E! [outputs.cloudwatchlogs] Aws error received when sending logs to ./audit: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: socket: too many open files
```
**Solution**:
Detect the newest changes when tailing and compare current time. If there was nothing changes regarding the log file, kill the tail process.
**Alternative solution:**
* [Increase max open file limit](http://woshub.com/too-many-open-files-error-linux/) on a particular OS
* Use [Log Sweeper](https://github.com/umbrellio/log_sweeper) to rotate and delete logs after a certain amount of time.
Contributor guide
Research direction
No file or test is named. Start by tracing the log tailing path and how inactivity and auto_removal are currently handled; confirm how unchanged files accumulate monitored processes and file descriptors. Done means inactive log files stop being tailed after the defined period without affecting updated files, and the too-many-open-files failure is prevented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100