aws / aws/containers-roadmap

[AWS Batch] awsfirelens Log Driver Support in AWS Batch with Fargate

Open
#2,573 9 comments 0 reactions 0 assignees View on GitHub
Fargate
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

I'm trying to configure awsfirelens as the log driver for my AWS Batch jobs running on Fargate, similar to the approach described in this [article](https://docs.datadoghq.com/integrations/ecs_fargate/?tab=webui#fluent-bit-and-firelens). According to the documentation, it appears that awsfirelens is not currently supported for AWS Batch.

However, when creating a JobDefinition (type: Container, orchestration type: Fargate) via the AWS Console, I am able to configure both the awsfirelens log driver and the FirelensConfiguration block without issue. Please see the attached screenshots for reference:

Image
Image

When attempting to replicate this configuration via the API (using Terraform), I receive the following error:

`Error: updating Batch Job Definition (): operation error Batch: RegisterJobDefinition, https response error StatusCode: 400, RequestID: 04e3cbdb-ba02-456a-9e65-caf583224da6, ClientException: Error executing request, Exception : When awsfirelens log driver is specified in log configuration, a firelens configuration object must be configured in one of the containers, RequestId: 04e3cbdb-ba02-456a-9e65-caf583224da6`

This suggests the awsfirelens log driver shouldn't be available, yet it seems to work when set up via the Console. I'm wondering:

Is this an unsupported configuration that happens to work through the Console?

Is this a bug or an upcoming feature that’s being progressively rolled out?

Are there any plans to officially support awsfirelens in AWS Batch on Fargate?

I currently use awsfirelens in my ECS services and would greatly benefit from having consistent logging across services, including Batch.

Any clarification or updates would be greatly appreciated.

Thank you!

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the configuration through Terraform and the AWS Batch RegisterJobDefinition API, then compare it with the AWS Console setup and the documented AWS Batch/Fargate support. Done means the issue has a clear answer on whether awsfirelens and FirelensConfiguration are supported, a bug, or planned, with any required support-status or documentation update identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, terraform
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.