aws / aws/amazon-cloudwatch-agent
Parse epoch time from log line
- Dominant language
- Go
- Stars
- 550
- Forks
- 271
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
Currently our log files contain a timestamp in epoch time for each line. There is no option for parsing epoch time using the `timestamp_format` property (https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Logssection). As a result, the CloudWatch Agent is using the time of parsing as the timestamp which could be quite different from the logged timestamp. This means needing to rely on custom sorting in CloudWatch Insights and makes formatting the timestamp more challenging.
**Describe the solution you'd like**
A mechanism to specify that the agent should use the timestamp in the log files (via `timestamp_format` or other means).
We are using structured JSON logs (as hopefully many are these days), so parsing out the timestamp could be done using a JSON parser instead of timestamp format regexes.
Even just a default where if the (JSON) structured log contained a property called "timestamp" or "time" that would be used as the logged timestamp. Using the parsing time is as the default timestamp is just really imprecise.
**Describe alternatives you've considered**
The alternative would be to generate an ISO8601 formatted timestamp in our log files and then parse it using the CloudWatch Agent - but this feels really suboptimal as the CloudWatch Agent is just converting it back into a timestamp after parsing. It would be preferable to be able to parse and use the epoch time directly from the log file entry.
The other option is to bypass the CloudWatch Agent altogether and post LogEvents directly to CloudWatch, but this feels like overkill.
Contributor guide
Assessment
This issue has not been assessed yet.