[EKS Fargate Logging] [request]: Document Message Line Limits and Add Support for >16kb
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
[EKS Fargate Logging](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html) currently appears to support a maximum length of 16kb per logged line. I request that this maximum length:
1. Be documented in a way that is both accurate and updated as it changes over time.
2. Be configurable to a value larger than 16kb such that large Json messages do not get split into two or more messages. A maximum of 32kb or 64kb would seem more appropriate.
**Which service(s) is this request for?**
This is for EKS Fargate Logging. Specifically, the behavior described in [this document](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html).
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I have a Spring Boot service written in Kotlin and I wanted to log its output in a "[json lines](https://jsonlines.org/)" format (one Json object per line) by leveraging [the common `JsonLayout` Log4j configuration](https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout). When exceptions are thrown and logged within the service, the stack trace is usually large enough that the resulting block of Json can be over 16kb. The EKS Fargate Logging worker splits this message in two, leaving the message in two strings that could not be parsed as Json, preventing it from being filtered upon downstream in a log viewing tool such as CloudWatch or Kibana. It's hard to filter within logging tools find these logs as you can't filter on its content via well formed fields as the Json didn't get parsed. Even if you do find the message, you have to manually stitch the messages back together to find out what happened.
**Are you currently working around this issue?**
I swapped our log4j configuration from [`JsonLayout`](https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout) to [`JsonTemplateLayout`](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html). The latter has a configurable `maxStringLength` attribute and can "stringify" stack traces to they get emitted as a single string. When I set the `maxStringLength` to `10000` and set stack traces with `stringified: true`, the stack traces are now truncated when they are large enough to trigger the splitting behavior. Since none of the other fields seem to total to more than ~6000 characters combined, the splitting of large messages has stopped.
**Additional context**
According to AWS documentation, EKS Fargate Logging is using FluentBit and generates its own `[Input]` blocks ([Source](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html) **(emphasis mine)**):
> ##### Validation Strategy
>
> The main sections included in a typical `Fluent Conf` are `Service`, `Input`, `Filter`, and `Output`. **`Service` and `Input` are generated by `Fargate`.** `Fargate` only validates the `Filter`, `Output`, and `Parser` specified in the `Fluent Conf`. Any sections provided other than `Filter`, `Output`, and `Parser` are ignored.
I believe these messages are running into the Docker daemon's internal/hardcoded 16kb limit for logged message before it flushes. The docker maintainers expect log parsing tools, such as Fluent Bit, to stitch these piecemeal messages back together again. Fluent Bit actually has an option to do this within the `Input` blocks using `docker_mode` ([Source](https://docs.fluentbit.io/manual/pipeline/inputs/tail#docker_mode)):
> ##### Docker Mode Configuration Parameters
>
> Docker mode exists to recombine JSON log lines split by the Docker daemon due to its line length limit. To use this feature, configure the tail plugin with the corresponding parser and then enable Docker mode:
>
> | Key | Description | Default |
> | ------------- | ------------- | ---- |
> | `Docker_Mode` | If enabled, the plugin will recombine split Docker log lines before passing them to any parser as configured above. This mode cannot be used at the same time as Multiline. | Off |
So I'm guessing the `Input` blocks generated from EKS Fargate Logging do not have `docker_mode` enabled. Assuming it is enabled, we'll run into limits around the size of `Buffer_Chunk_Size` (32kb by default) eventually as well. I have not observed logs being generated over around ~20kb from our service though, so that limit would at least be sufficient for us.
**Attachments**
I've attached three things:
1. An `aws-logging.yaml` file that maps to the `ConfigMap` used to parse EKS Fargate logs.
2. An `example.json` JSON log file that the service emitted which is over 16kb.
3. The end result of the EKS Fargate `ConfigMap` being applied to a JSON log over 16kb being broken into `example-split-first-half.json` and `example-split-second-half.json`.
[examples.zip](https://github.com/aws/containers-roadmap/files/6209366/examples.zip)
Contributor guide
Research direction
Review the linked EKS Fargate Logging and Fluent Bit documentation, then inspect the attached aws-logging.yaml and example JSON files to reproduce the split-message behavior. Done means the supported line limit is accurately documented and larger JSON log messages are handled without being split, if the service supports that change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, kotlin, kubernetes, spring
- Domain
- cloud, devops, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100