kestra-io / kestra-io/plugin-dbt
Add dbt models and tests to the Execution Topology view (the same as currently shown in the Gantt view)
- Dominant language
- Java
- Stars
- 14
- Forks
- 14
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 10
Description
Extend the `dbtRun`, `dbtBuild` and `dbtCll` tasks with 2 new properties parse and `outputRunResults`:
- [ ] allow rendering dbt dag at runtime in the Execution topology view if `parse` (new property) is set to `true`
- [ ] add another boolean flag `outputRunResults` indicating whether to output dbt models with custom metadata like duration, which will allow later to expose those as events — ideally, an ION file with one row per dbt model or test (data can be extracted from `run_results.json` generated by dbt after dbt Build/Run completion)
Name | Type | Status | Duration | Completed At
-- | -- | -- | -- | --
model.dwh.stg_customers | Model | success | 0.022 | 2024-03-25T08:46:21.888471Z
model.dwh.stg_orders | Model | success | 0.022 | 2024-03-25T08:46:21.888663Z
model.dwh.stg_locations | Model | success | 0.023 | 2024-03-25T08:46:21.889137Z
model.dwh.stg_order_items | Model | success | 0.023 | 2024-03-25T08:46:21.889313Z
model.dwh.stg_products | Model | success | 0.023 | 2024-03-25T08:46:21.890272Z
model.dwh.stg_supplies | Model | success | 0.023 | 2024-03-25T08:46:21.890450Z
test.dwh.not_null_stg_customers_customer_id | Test | success | 0.042 | 2024-03-25T08:46:21.923868Z
test.dwh.unique_stg_customers_customer_id | Test | success | 0.044 | 2024-03-25T08:46:21.924386Z
test.dwh.not_null_stg_orders_order_id | Test | success | 0.044 | 2024-03-25T08:46:21.935854Z
test.dwh.unique_stg_orders_order_id | Test | success | 0.046 | 2024-03-25T08:46:21.938104Z
## Context
dbt models and tests are difficult to read and inspect in the Gantt view:

We would like to see those in the Execution Topology view:

Simple reproducer to inspect/iterate on the design (you can run it directly with no setup):
```yaml
id: dbtGitDockerDuckDB
namespace: dev
tasks:
- id: dbt
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-demo
branch: main
- id: dbt-build
type: io.kestra.plugin.dbt.cli.Build
debug: true
dockerOptions:
image: ghcr.io/kestra-io/dbt-duckdb:latest
runner: DOCKER
dbtPath: /usr/local/bin/dbt
inputFiles:
.profile/profiles.yml: |
jaffle_shop:
outputs:
dev:
type: duckdb
path: ':memory:'
extensions:
- parquet
target: dev
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start from the dbtRun, dbtBuild, and dbtCll task entry points and use the supplied dbtGitDockerDuckDB reproducer to inspect the current Gantt and Execution Topology behavior. Done means parse can enable runtime dbt DAG rendering and outputRunResults can expose one row per dbt model or test using run_results.json data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100