opensearch-project / opensearch-project/data-prepper
Poll CloudWatch for metrics as a source
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Pipeline authors want to get metrics out of Amazon CloudWatch to send to either OpenSearch or Prometheus.
Describe the solution you'd like
Create a CloudWatch source plugin which retrieves metrics from CloudWatch. It should put those metrics into the Data Prepper in the current Metrics model, which is based on the OpenTelemetry specification.
source:
cloudwatch:
namespace: ES/OpenSearchService
metric_name: ShardCount
dimensions:
- name: DomainName
value: my-domain
As this is a polling source, it should also allow pipeline authors to define a poll interval similar to the S3 source.
It also needs to have AWS configurations. These can follow similar conventions to other AWS-based plugins.
Describe alternatives you've considered (Optional)
An alternative design would be to put the metrics into the buffer in the CloudWatch format. Then pipeline authors could configure a processor which performs the translation. This approach could be valuable if the translation is non-trivial and requires some additional configurations from pipeline authors. It could also be valuable if the translation is costly and we want to have it happen on data in the buffer.
Additional context
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests; start by reviewing the existing S3 source and other AWS-based plugins, then read the CloudWatch GetMetricStatistics API documentation. Done would mean a configurable polling source retrieves the requested CloudWatch metrics, emits them in the current OpenTelemetry-based Metrics model, and supports the stated AWS configuration conventions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100