Cloudwatch extractor memory issue while writing logs to file
- Dominant language
- Python
- Stars
- 17
- Forks
- 13
- Avg merge
- 8h 23m
- Merged PRs (30d)
- 1
Description
I am working with a customer to do node comparison for a large cluster. They are using CloudWatch to capture audit logs.
The extractor task failed with "Memory Error" while writing CloudWatch logs to a zip while. The parser then reads this file and generates the query workload to replay.
There are following improvements which can be done to the tool -
1. Provide input parameters to configure memory, CPU and storage. Also provide documentation to follow the steps to update memory, CPU and storage if the task fails.
2. Optimized the code such that there is no need to write CloudWatch logs to a local file , read from there and parse to generate workload. There is no need to write file to local storage.
Contributor guide
Research direction
The issue names no files or tests; start by tracing the CloudWatch extractor task through the parser that generates the replay workload. Confirm where writing logs to a zip causes the memory error, then scope whether the work is resource configuration and documentation or avoiding the local file; done means the large-cluster workload can be generated without the reported failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100