awslabs / awslabs/aws-embedded-metrics-node

Add a CircularBuffer in AgentSink

Open
#36 3 comments 1 reaction 0 assignees View on GitHub
enhancement in progress
Dominant language
TypeScript
Stars
263
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Description

Currently, if the agent is down or has not started, metrics can be dropped. It's currently up to the caller of `logger.flush` to handle retries. There are 2 options:

1. Backpressure the caller of `logger.flush`. This could negatively impact request latencies.
2. On error, enqueue to a circular buffer. The trick here is we will need to retry this queue on an interval which changes the model from an async/await to a purely async one. This is a departure from the current design and will need to be turned on via feature flag.

The symptoms of this are:
1. The first metrics during initialization of the app may not appear
2. The following error message will be in your app logs:

```
(node:1) UnhandledPromiseRejectionWarning: Error: connect ECONNREFUSED 172.17.0.2:25888
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
```

## Tasks

- Add type `AgentSinkOptions` with
- `RetryStrategy` parameter where the default value is `None` for backwards compatibility with a single option to start with: `ExponentialBackoffRetryStrategy` (see also: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/)
- `AsyncBehavior` parameter that controls whether the call should block or not. In the former case we keep the [current behavior](https://github.com/awslabs/aws-embedded-metrics-node/blob/master/src/sinks/connections/TcpClient.ts#L62) and in the latter we return immediately, enqueuing to the retry buffer on failure.
- Change AgentSink's constructor to `constructor(options: AgentSinkOptions, ISerializer: serializer)`.
- Add RetryStrategies which the AgentSink uses based on its configuration. `NoRetry` propagates errors back to the caller of `flush` which maintains current behavior today. `ExponentialRetry` (which can be configured by the application) will block `flush` on the first attempt, enqueuing to a `CircularBuffer` (whose size is also configurable) on failures.
- On startup, `setInterval` will be set to check the size of the `CircularBuffer` and retry failed requests asynchronously.
- Add `shutdown` method to gracefully shutdown and block on any outstanding requests.

## Example Usage

```js
AWS_EMF_AGENT_RETRY_STRATEGY="ExponentialBackoff"
// or
Configuration.agentRetryStrategy = RetryStrategy.ExponentialBackoff;
// or
Configuration.agentRetryStrategy = (...) => customRetryStratgy();

// ...
await logger.flush();
// execution control is returned when logs have been successfully flushed or enqueued for retry
```

## Open Question

- Should we change `logger.flush()` to enqueue and return immediately? This would allow us to make `flush()` a synchronous operation in all cases.

Contributor guide

Open the contributing guide

Research direction

Start by reading src/sinks/connections/TcpClient.ts at line 62 and tracing AgentSink and logger.flush. Map the requested AgentSinkOptions, retry strategies, CircularBuffer, interval processing, and shutdown behavior before resolving the open question about flush semantics. Done means the configured retry behavior, bounded buffering, asynchronous retries, and graceful shutdown are implemented without changing the default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.