aws / aws/aws-cdk

Real unit calculations for CloudWatch

Open
#5,595 1 comment 12 reactions 0 assignees View on GitHub
@aws-cdk/aws-cloudwatch effort/medium feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

Right now, the `unit` field in CloudWatch does not do what one would expect. We can make the CDK cover this usability gap though.

----

Metrics emitted to CloudWatch can have *units*, such as as `seconds`, `bytes`, `bytes/second`, etc. For example, a service can emit the metrics (in order):

```
1000000 bytes
2000000 bytes
3 megabytes
```

**Alarms** in CloudWatch also have *units*. However, it does **not** rescale the value in the alarm, or the values in the metrics. In fact, what happens is that it **filters** for metrics that have been emitted with the same unit as the one set on the alarm. So if you set the alarm to `1 megabyte`, it will trigger on the 3rd datapoint emitted in the previous example (the 2 metrics emitted with unit `bytes` will be ignored).

Source: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Statistic

> Though CloudWatch attaches no significance to a unit internally, other applications can derive semantic information based on the unit.
>
> If you have two otherwise identical metrics with different units, two separate data streams are returned, one for each unit.

----

Although it makes sense given implementation details of CloudWatch, I would argue that this interpretation of the property `unit` is **not what one would expect to happen**. All 3 metrics are a measure of the same *quantity* (which is "size"), and conceptually I would like to be able to say "an alarm with a threshold of 1MB", regardless of the exact unit with which the metrics are being emitted.

Note that the feature where you configure a `unit` for an Alarm is intended to correct for cases where the same metric is being emitted with different units. However, this use case itself is not recommended (see *Unit* section [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricAlarm.html#API_PutMetricAlarm_RequestParameters)), and AFAIK, no AWS services are doing this.

-----

It occurred to me that we can be doing the unit calculations for users, to make the common use case work. It does require adding a `baseUnit` (or whatever) field to `Metric`, which is the physical unit that values are emitted under. We would need to enhance all default Metrics objects generated by the CDK (such as [these ones for CodeBuild](https://github.com/aws/aws-cdk/blob/3256a41787c365a67b01bee193bd75e48645f7a0/packages/%40aws-cdk/aws-codebuild/lib/project.ts#L106-L149)) with the actual units that are emitted by the services into CloudWatch. This would be human effort, this information is not captured anywhere as far as I know.

Obviously, the default needs to be `Unknown` and sensible stuff needs to happen if that is the value of the metric (which would be the same behavior as today).

Once we know about the actual units of metrics in CloudWatch, we can do the following:

* Automatically rescale alarm values to the right value (in the example, the user would configure an alarm for `1MB` but we'd create the actual alarm for `1000000 bytes`).
* Same for annotations in graphs.
* Automatically label the Y axis in dashboard graphs with the appropriate unit (`seconds`, `milliseconds`, `hours`).
* We can error (warn?) if you add multiple metrics of different quantities to the same graph (adding both `CpuUtilization` and `NetworkInBytes` on the left axis? That probably doesn't make any sense)
* Graph axis ranges can be specified in any unit as well (scale from `10 bytes..10MB`).
* Using the new Metric Math feature we can have the graph automatically rescaled: If we set the `yAxis` to be `MB`, we can automatically wrap all metrics added to it that are known to be in `bytes` into a Math expression that does `M1 / 1000000`.

-----

Obviously, us doing this will not magically change the behavior of CloudWatch, and we'd still need to expose the "filtering" behavior of CloudWatch's `unit` field in some way, but it would be a more useful and intuitive implementation of unit than what we currently have.

Alternatively, we repurpose the current `unit` field to our unit calculations, and we add a new field `unitFilter` (or whatever) to expose CloudWatch's native behavior.

-----

Implementation-wise, I would be thrilled if the CDK core team were able to pick this up, but we will probably be focusing on other things in the near future. Hoping for community contributions on this one.

---

This is a :rocket: Feature Request

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Metric model and the default metrics in packages/@aws-cdk/aws-codebuild/lib/project.ts. Trace how alarms, graph annotations, dashboard axes, and metric math currently use units. Done would require an agreed unit representation and consistent handling of scaling, filtering, and unknown units across the proposed cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.