kestra-io / kestra-io/plugin-dbt

Add OTEL trace capture via koltp for BigQuery/warehouse queries

Open
#277 0 comments 0 reactions 1 assignee Claimed by @Malaydewangan09 View on GitHub
area/docs area/plugin
Dominant language
Java
Stars
14
Forks
14
Avg merge
1d 10h
Merged PRs (30d)
10

Description

## Context

[koltp](https://github.com/kestra-io/koltp) is a cross-platform process wrapper that captures OTEL traces, metrics (CPU/mem/network/IO), and structured logs from any process.

When a task runner has `monitoring.enabled: true`, koltp wraps the process and forwards OTEL data back to Kestra via `::{"oltp":}::` markers on stdout.

## Goal

dbt natively supports OpenTelemetry via its `--log-format` and the [dbt-opentelemetry](https://github.com/dbt-labs/dbt-core/discussions/7169) initiative. When koltp starts the dbt process with an OTEL collector endpoint configured, all queries sent to the data warehouse (BigQuery, Snowflake, Redshift, etc.) will be captured as OTEL spans.

## Work

- Validate that dbt emits OTEL traces when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (provided by koltp)
- Document what spans/attributes are captured (model name, query, duration, warehouse)
- If dbt's native OTEL support is incomplete, add a post-processor or log parser to extract query-level spans
- Add documentation explaining what is visible when `taskRunner.monitoring.enabled: true`

## Value

- Every SQL query sent to the warehouse visible as a trace inside Kestra
- Query duration trending over time (e.g. "this dbt model took 30min last month, now 2h")
- Stored in Kestra's trace store + forwarded to external OTEL collector if configured

## Related

- koltp: https://github.com/kestra-io/koltp
- System usage tracking: https://github.com/kestra-io/kestra/issues/310
- Task runner monitoring prop (to be added per task runner)
- Implementation challenge: https://app.notion.com/p/kestra-io/Implementation-challenge-37c36907f7b580ea9af9c86569af20b3

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.