awslabs / awslabs/aws-embedded-metrics-node

[Feature] [development] new configuration to filter properties out of the serialised JSON log on the stdout.

Open
#57 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
263
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Describe the user story
Currently, when I set `AWS_EMF_ENVIRONMENT=Local` and run the app, the agent logs to the stdout which is the expected behaviour.

For small logs this is fine, but when it comes to bigger projects the local sandbox terminal is flooded with tons of serialised mammoth JSON objects.

As a developer, I'd like to propose a new configuration that defines which properties of the JSON object can be logged by the agent so that I can dynamically customise the amount of details printed on the stdout. This functionality should be only available in development (i.e. when `AWS_EMF_ENVIRONMENT` is set to `Local`).

## Use case

Let's say we have a log structure like the following:

```json
{
"PageType": "player",
"event": {
"id": "9bac0a47-1623-410d-bcd2-f03aa1283669",
"source": "server",
"trigger": "user",
"type": "page-request"
},
"logTrace": [
"[UpstreamName.apifeed]: Empty data, returning with graceful degradation."
],
"processPid": 872,
"requestPath": "/path/to/the/resource",
"requestHeaders": {

},
"requestHeadersList": [

],
"hasCookie": false,
"cookieLength": 0,
"cookieList": [],
"responseStatus": 200,
"responseHeaders": {

},
"upstreams": [
{
"name": "UpstreamName",
"endpoint": "apifeed",
"attempts": [
{
"cache": {
"hit": true,
"miss": false,
"stale": false,
"error": false,
"timeout": false,
"revalidate": false,
"revalidateError": false
},
"response": {
"headers": {

},
"body": {

},
"status": 200,
"time": 86
},
"id": 1
}
],
"attemptCount": 1,
"retryCount": 0,
"requestCount": 0,
"requestErrorCount": 0,
"response5xxCount": 0,
"response4xxCount": 0,
"response3xxCount": 0,
"response2xxCount": 0,
"response1xxCount": 0,
"responseInvalidCount": 0,
"cacheAudit": [
[
"hit"
]
],
"cacheHitCount": 1,
"cacheMissCount": 0,
"cacheStaleCount": 0,
"cacheErrorCount": 0,
"cacheTimeoutCount": 0,
"cacheRevalidateCount": 0,
"cacheRevalidateErrorCount": 0,
"responseTime": 10
}
],
"imageId": "ami-someid",
"instanceId": "i-someid",
"instanceType": "some.instancetype",
"privateIP": "127.0.0.1",
"availabilityZone": "some-aws-region",
"_aws": {
"Timestamp": 2693848470655,
"LogGroupName": "/example/live/player/app",
"CloudWatchMetrics": [
{
"Dimensions": [
[
"PageType"
]
],
"Metrics": [
{
"Name": "RequestCount",
"Unit": "Count"
},
{
"Name": "ResponseTime",
"Unit": "Milliseconds"
},
{
"Name": "ErrorCount",
"Unit": "Count"
},
{
"Name": "PageNotFoundCount",
"Unit": "Count"
}
],
"Namespace": "/example/live/player/app"
}
]
},
"RequestCount": 1,
"ResponseTime": 211,
"ErrorCount": 0,
"PageNotFoundCount": 0
}
```

and I'm running the app locally. This object is clearly big to print out and if you think that it is serialised and logged once every time a new requests is performed by the user, you can imagine how busy the terminal will look like.

**[UPDATE] the following section has been "quoted" to highlight that an amendment of the following requests has been added in the comments. The section remains here in the description for visibility.**

> Let's say I only want to log some textual information and not everything else. If you notice, among all these useful in production (but noisy in development) properties, there is an array called `logTrace`, I'd like to be able to do something similar to:
>
> ```js
> // in process
> const { Configuration } = require("aws-embedded-metrics");
> Configuration.somePropertyName = ['logTrace'];
>
> // environment
> AWS_EMF_SOME_PROPERTY_NAME="logTrace"
> ```
>
> and on the terminal, printing something like the following:
>
> ```bash
> { "logTrace": [ "[UpstreamName.apifeed]: Empty data, returning with graceful degradation." ] }
> ```
>

> ## Details
>
> The new configuration should have the following requirements:
> * We can define multiple properties (comma separated for the environment variable, or using an array for the in-code variable)
> * We can define nested properties _a-la_ [lodash#get](https://lodash.com/docs/4.17.15#get) (e.g. "event.type") by using the dot-notation.
> * The agent will print a flattened object where all properties appear at the root and the key is the name of the property. If a selected key is nested (e.g. `event.type`) the property will use the dot-notation as a key name.
> * The new configuration is only active when `AWS_EMF_ENVIRONMENT=Local` so that it only apply in development.
> * If only one property is selected and its value is a string, only log the string. This way if you have a property that you use to log as you used to do before it will be pretty much similar to what you had.
>
> I'm not particularly opinionated on the name of the config. I can give a couple examples but I'm open to other suggestions: `AWS_EMF_ALLOWED_PROPERTIES`, `AWS_EMF_FILTERED_PROPERTIES`, `AWS_EMF_LOCAL_LOG_STRUCTURE`, `AWS_EMF_LOCAL_LOG_PROPERTIES` (or some permutation). The last two environment variables suggest what they are about, and only work locally.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Configuration API and the code path that serialises and prints logs when AWS_EMF_ENVIRONMENT is Local. Trace how environment and in-code settings are read, then define tests for multiple and nested properties, flattened output, string-only output, and unchanged non-local behaviour; done means those cases are covered and the terminal output follows the selected configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
backend, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.