Structured log messages
- Dominant language
- Go
- Stars
- 58
- Forks
- 55
- Avg merge
- 9h 9m
- Merged PRs (30d)
- 424
Description
### Motivation
Every log Cloudbeat outputs is a JSON object in a `.ndjson` file.
On managed elastic stacks, filebeat reads these log files and indexes them into elasticsearch, this data can be the source of interesting aggregations which will later be used for visualizations and alerts to allow proactive monitoring.
One issue we currently have is that many of the logs are formatted, meaning parameters are part of the log message instead of introducing them in a dedicated field of this json object.
This change is effectively done by changing the logging calls from:
`log.Errorf("Could not get encryption for bucket %s. Error: %v", *bucket.Name, encryptionErr)`
into:
`log.Errorw("Could not get encryption for S3 bucket", "bucket", *bucket.Name, "error", encryptionErr)`
This change brings another challenge in the area of naming the new fields. With this change "bucket" and
"error" are now keys in the json object that indexed to ES and therefore should comply with ECS.
There is not an easy way to name those fields, we need to individually assess every log and find the best matching ECS fields that represent its parameters. In the example about, a better representation will be:
`log.Errorw("Could not get encryption for S3 bucket", "aws.s3.bucket.name", *bucket.Name, "error.message", encryptionErr)`
### Definition of done
- [ ] Refactor cloudbeat to have structured logs instead of formatted logs where possible
### Related tasks/epics
-
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.