aws / aws/aws-cdk

[ecs-patterns] Simplify JSON Line Log Message Parsing with Firelens and ApplicationLoadBalancedFargateService

Open
#10,813 3 comments 5 reactions 0 assignees View on GitHub
@aws-cdk/aws-ecs-patterns effort/small feature-request feature/pattern p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

Provide documentation for and/or configuration of a property related to JSON line log parsing and/or expose the created firelensLogRouter so its configuration can be modified.

### Use Case
I have a SpringBoot application that logs JSON lines using Logback and the encoder `net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder`

My log store is Sumo Logic.

Log messages appear as escaped strings that are not readable, when using the default configuration provided by ApplicationLoadBalancedFargateService and a firelensLogDriver.
![image](https://user-images.githubusercontent.com/136476/95669061-17388e80-0b4a-11eb-84d6-f8bba4754f7c.png)

The firelensLogDriver does not provide a way to set the `FirelensConfig.options` of the `FirelensLogRouter` that ultimately is created. The `FirelensLogRouter` is not exposed as a property of ApplicationLoadBalancedFargateService. The `TaskDefinition` exposes the `defaultContainer` (my application) but the `containers` property of `TaskDefinition` is protected so the there is no way to obtain the `FireLensLogRouter` container and modify it.

Ultimately, after long searches of source code and custom logging documentation, I was able to override the FirelensLogRouter.

```typescript
protected _createFargateService() {
const logDriver = LogDrivers.firelens({
options: {
Name: 'http',
Host: this._props.containerLogging.endpoint,
URI: this._props.containerLogging.uri,
Port: '443',
tls: 'on',
'tls.verify': 'off',
Format: 'json_lines'
}
});
const fargateService = new ApplicationLoadBalancedFargateService(this, this._props.serviceName, {
cluster: this._accountEnvironmentLookups.getComputeCluster(),
cpu: this._props.cpu, // Default is 256
desiredCount: this._props.desiredCount, // Default is 1
taskImageOptions: {
image: ContainerImage.fromEcrRepository(this._props.serviceRepository, this._props.imageVersion),
environment: this._props.environment,
containerPort: this._props.containerPort,
logDriver
},
memoryLimitMiB: this._props.memoryLimitMiB, // Default is 512
publicLoadBalancer: this._props.publicLoadBalancer, // Default is false
domainName: this._props.domainName,
domainZone: !!this._props.hostedZoneDomain ? HostedZone.fromLookup(this, 'ZoneFromLookup', {
domainName: this._props.hostedZoneDomain
}) : undefined,
certificate: !!this._props.certificateArn ? Certificate.fromCertificateArn(this, 'CertificateFromArn', this._props.certificateArn) : undefined,
serviceName: `${this._props.accountShortName}-${this._props.deploymentEnvironment}-${this._props.serviceName}`,
// The new ARN and resource ID format must be enabled to work with ECS managed tags.
//enableECSManagedTags: true,
//propagateTags: PropagatedTagSource.SERVICE,
// CloudMap properties cannot be set from a stack separate from the stack where the cluster is created.
// see https://github.com/aws/aws-cdk/issues/7825
});
if (this._props.logMessagesAreJsonLines) {
// The default log driver setup doesn't enable json line parsing.
const firelensLogRouter = fargateService.service.taskDefinition.addFirelensLogRouter('log-router', {
// Figured out how get the default fluent bit ECR image from here https://github.com/aws/aws-cdk/blob/60c782fe173449ebf912f509de7db6df89985915/packages/%40aws-cdk/aws-ecs/lib/base/task-definition.ts#L509
image: obtainDefaultFluentBitECRImage(fargateService.service.taskDefinition, fargateService.service.taskDefinition.defaultContainer?.logDriverConfig),
essential: true,
firelensConfig: {
type: FirelensLogRouterType.FLUENTBIT,
options: {
enableECSLogMetadata: true,
configFileType: FirelensConfigFileType.FILE,
// This enables parsing of log messages that are json lines
configFileValue: '/fluent-bit/configs/parse-json.conf'
}
},
memoryReservationMiB: 50,
logging: new AwsLogDriver({streamPrefix: 'firelens'})
});
firelensLogRouter.logDriverConfig;
}
fargateService.targetGroup.configureHealthCheck({
path: this._props.healthUrlPath,
port: this._props.containerPort.toString(),
interval: Duration.seconds(120),
unhealthyThresholdCount: 5
});
const scalableTaskCount = fargateService.service.autoScaleTaskCount({
minCapacity: this._props.desiredCount,
maxCapacity: this._props.maxCapacity
});
scalableTaskCount.scaleOnCpuUtilization(`ScaleOnCpuUtilization${this._props.cpuTargetUtilization}`, {
targetUtilizationPercent: this._props.cpuTargetUtilization
});
scalableTaskCount.scaleOnMemoryUtilization(`ScaleOnMemoryUtilization${this._props.memoryTargetUtilization}`, {
targetUtilizationPercent: this._props.memoryTargetUtilization
});
this.fargateService = fargateService;
}
```

### Proposed Solution
At the very least, documentation regarding how to override the `FirelensLogRouter` so that custom Firelens configuration can be provided, would be extremely helpful. Documentation also suggests that regional registries should be used for obtaining the AWS Fluentbit ECR image, but the `obtainDefaultFluentBitECRImage` function is undocumented.

Since JSON line logging is very common, it would be helpful to be able to specify an option of the log driver that would enable the parsing configuration shown here, so the creation of the `FirelensLogRouter` is not necessary.

Absent such a configuration, exposing the created `FirelensLogRouter` might be helpful IF its configuration can be modified. Most resources have `readonly` properties so I imagine that might be difficult. Additionally, this resource may be getting created at render time, which is what allows me to override it in the first place, since a check is performed to see if the router is already present in the task definition.

This might require a bit of cross team discussion.

### Other
This stack overflow solution has additional resources that were used to come up with the code listed above. It could use a little ❤️
https://stackoverflow.com/questions/64299664/how-to-configure-aws-cdk-applicationloadbalancedfargateservice-to-log-parsed-jso

* [ ] :wave: I may be able to implement this feature request
* [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

Contributor guide

Open the contributing guide

Research direction

Start with the ecs-patterns ApplicationLoadBalancedFargateService entry point and the FirelensLogRouter creation path, then review the referenced obtainDefaultFluentBitECRImage implementation and existing documentation. Done should be a settled, documented or exposed configuration path for JSON-line parsing, with the behavior and customization limits made clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.