aws / aws/redshift-test-drive

Cloudwatch extractor memory issue while writing logs to file

Open
#130 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
17
Forks
13
Avg merge
8h 23m
Merged PRs (30d)
1

Description

I am working with a customer to do node comparison for a large cluster. They are using CloudWatch to capture audit logs.
The extractor task failed with "Memory Error" while writing CloudWatch logs to a zip while. The parser then reads this file and generates the query workload to replay.
There are following improvements which can be done to the tool -
1. Provide input parameters to configure memory, CPU and storage. Also provide documentation to follow the steps to update memory, CPU and storage if the task fails.
2. Optimized the code such that there is no need to write CloudWatch logs to a local file , read from there and parse to generate workload. There is no need to write file to local storage.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; start by tracing the CloudWatch extractor task through the parser that generates the replay workload. Confirm where writing logs to a zip causes the memory error, then scope whether the work is resource configuration and documentation or avoiding the local file; done means the large-cluster workload can be generated without the reported failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.