aws / aws/amazon-cloudwatch-agent

Parse epoch time from log line

Open
#683 0 comments 7 reactions 0 assignees View on GitHub
area/log enhancement
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
Currently our log files contain a timestamp in epoch time for each line. There is no option for parsing epoch time using the `timestamp_format` property (https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Logssection). As a result, the CloudWatch Agent is using the time of parsing as the timestamp which could be quite different from the logged timestamp. This means needing to rely on custom sorting in CloudWatch Insights and makes formatting the timestamp more challenging.

**Describe the solution you'd like**
A mechanism to specify that the agent should use the timestamp in the log files (via `timestamp_format` or other means).

We are using structured JSON logs (as hopefully many are these days), so parsing out the timestamp could be done using a JSON parser instead of timestamp format regexes.

Even just a default where if the (JSON) structured log contained a property called "timestamp" or "time" that would be used as the logged timestamp. Using the parsing time is as the default timestamp is just really imprecise.

**Describe alternatives you've considered**
The alternative would be to generate an ISO8601 formatted timestamp in our log files and then parse it using the CloudWatch Agent - but this feels really suboptimal as the CloudWatch Agent is just converting it back into a timestamp after parsing. It would be preferable to be able to parse and use the epoch time directly from the log file entry.

The other option is to bypass the CloudWatch Agent altogether and post LogEvents directly to CloudWatch, but this feels like overkill.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.