elastic / elastic/cloudbeat

Structured log messages

Open
#1,888 5 comments 4 reactions 1 assignee Assigned to @Copilot View on GitHub
cloudbeat Team:Cloud Security
Dominant language
Go
Stars
58
Forks
55
Avg merge
9h 9m
Merged PRs (30d)
424

Description

### Motivation
Every log Cloudbeat outputs is a JSON object in a `.ndjson` file.
On managed elastic stacks, filebeat reads these log files and indexes them into elasticsearch, this data can be the source of interesting aggregations which will later be used for visualizations and alerts to allow proactive monitoring.

One issue we currently have is that many of the logs are formatted, meaning parameters are part of the log message instead of introducing them in a dedicated field of this json object.
This change is effectively done by changing the logging calls from:
`log.Errorf("Could not get encryption for bucket %s. Error: %v", *bucket.Name, encryptionErr)`
into:
`log.Errorw("Could not get encryption for S3 bucket", "bucket", *bucket.Name, "error", encryptionErr)`

This change brings another challenge in the area of naming the new fields. With this change "bucket" and
"error" are now keys in the json object that indexed to ES and therefore should comply with ECS.
There is not an easy way to name those fields, we need to individually assess every log and find the best matching ECS fields that represent its parameters. In the example about, a better representation will be:
`log.Errorw("Could not get encryption for S3 bucket", "aws.s3.bucket.name", *bucket.Name, "error.message", encryptionErr)`

### Definition of done
- [ ] Refactor cloudbeat to have structured logs instead of formatted logs where possible

### Related tasks/epics
-

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.