kestra-io / kestra-io/plugin-dbt
Add OTEL trace capture via koltp for BigQuery/warehouse queries
- Dominant language
- Java
- Stars
- 14
- Forks
- 14
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 10
Description
## Context
[koltp](https://github.com/kestra-io/koltp) is a cross-platform process wrapper that captures OTEL traces, metrics (CPU/mem/network/IO), and structured logs from any process.
When a task runner has `monitoring.enabled: true`, koltp wraps the process and forwards OTEL data back to Kestra via `::{"oltp":}::` markers on stdout.
## Goal
dbt natively supports OpenTelemetry via its `--log-format` and the [dbt-opentelemetry](https://github.com/dbt-labs/dbt-core/discussions/7169) initiative. When koltp starts the dbt process with an OTEL collector endpoint configured, all queries sent to the data warehouse (BigQuery, Snowflake, Redshift, etc.) will be captured as OTEL spans.
## Work
- Validate that dbt emits OTEL traces when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (provided by koltp)
- Document what spans/attributes are captured (model name, query, duration, warehouse)
- If dbt's native OTEL support is incomplete, add a post-processor or log parser to extract query-level spans
- Add documentation explaining what is visible when `taskRunner.monitoring.enabled: true`
## Value
- Every SQL query sent to the warehouse visible as a trace inside Kestra
- Query duration trending over time (e.g. "this dbt model took 30min last month, now 2h")
- Stored in Kestra's trace store + forwarded to external OTEL collector if configured
## Related
- koltp: https://github.com/kestra-io/koltp
- System usage tracking: https://github.com/kestra-io/kestra/issues/310
- Task runner monitoring prop (to be added per task runner)
- Implementation challenge: https://app.notion.com/p/kestra-io/Implementation-challenge-37c36907f7b580ea9af9c86569af20b3
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.