[Feature] Add Custom Schema's for structured logs
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
It would be very handy to be able to convert the standard structured logs to another schema. This would be great to move away from the standard text format. For example, say I want to integrate all my services logs in grafana I would aim to have all logs produced by all services conform to one schema. This way I can index elements/ keys to allow for searches to be conducted against my logs - setting tags, alerting on specific log levels (errors) etc.
### Describe alternatives you've considered
I have tried to use the dbt core codebase to invoke a DBT run via a python script and set a callback to transform the logs, this mostly works however, for reasons I dont understand, I cannot:
- Stop some logs coming through in the normal DBT format - I get duplicates for random logs in the default format and my transformed format.
- stop debug logs coming through.
```
import json
from dbt.cli.main import dbtRunner
from dbt_common.events.base_types import EventMsg
from dbt_common.events.functions import msg_to_json
class EcsLogTransformer:
def __init__(self, ecs_version="1.0.0"):
self.ecs_version = ecs_version
def transform(self, event: EventMsg):
event_dict = json.loads(msg_to_json(event))
info = event_dict.get("info")
ecs_event = {
...
}
return ecs_event
def transform_to_ecs_structured_logs(event: EventMsg):
log_transformer = EcsLogTransformer()
ecs_event = log_transformer.transform(event)
print(ecs_event)
if __name__ == "__main__":
args = ["--log-format", "json", "--log-level", "info", "run"]
dbt = dbtRunner(callbacks=[transform_to_ecs_structured_logs])
dbt.invoke(args)
```
### Who will this benefit?
This would be useful for anyone who wants to set their own structure to the json log format. I am aware that many users of DBT have this set up on orchestrators etc. via k8's. Anyone who uses this to run dbt could benefit from their own structuring of logs.
### Are you interested in contributing this feature?
_No response_
### Anything else?
An example of another schema type: https://www.elastic.co/guide/en/ecs-logging/overview/current/intro.html
Contributor guide
Assessment
This issue has not been assessed yet.