dbt-labs / dbt-labs/dbt

[Feature] Add Custom Schema's for structured logs

Open
#11,381 2 comments 1 reaction 0 assignees View on GitHub
engine:v1 status:triage type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

It would be very handy to be able to convert the standard structured logs to another schema. This would be great to move away from the standard text format. For example, say I want to integrate all my services logs in grafana I would aim to have all logs produced by all services conform to one schema. This way I can index elements/ keys to allow for searches to be conducted against my logs - setting tags, alerting on specific log levels (errors) etc.

### Describe alternatives you've considered

I have tried to use the dbt core codebase to invoke a DBT run via a python script and set a callback to transform the logs, this mostly works however, for reasons I dont understand, I cannot:

- Stop some logs coming through in the normal DBT format - I get duplicates for random logs in the default format and my transformed format.
- stop debug logs coming through.

```
import json

from dbt.cli.main import dbtRunner
from dbt_common.events.base_types import EventMsg
from dbt_common.events.functions import msg_to_json

class EcsLogTransformer:
def __init__(self, ecs_version="1.0.0"):
self.ecs_version = ecs_version

def transform(self, event: EventMsg):
event_dict = json.loads(msg_to_json(event))
info = event_dict.get("info")

ecs_event = {
...
}
return ecs_event

def transform_to_ecs_structured_logs(event: EventMsg):
log_transformer = EcsLogTransformer()
ecs_event = log_transformer.transform(event)
print(ecs_event)

if __name__ == "__main__":
args = ["--log-format", "json", "--log-level", "info", "run"]
dbt = dbtRunner(callbacks=[transform_to_ecs_structured_logs])
dbt.invoke(args)
```

### Who will this benefit?

This would be useful for anyone who wants to set their own structure to the json log format. I am aware that many users of DBT have this set up on orchestrators etc. via k8's. Anyone who uses this to run dbt could benefit from their own structuring of logs.

### Are you interested in contributing this feature?

_No response_

### Anything else?

An example of another schema type: https://www.elastic.co/guide/en/ecs-logging/overview/current/intro.html

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.