kestra-io / kestra-io/plugin-dbt

Add dbt models and tests to the Execution Topology view (the same as currently shown in the Gantt view)

Open
#38 2 comments 1 reaction 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
14
Forks
14
Avg merge
1d 10h
Merged PRs (30d)
10

Description

Extend the `dbtRun`, `dbtBuild` and `dbtCll` tasks with 2 new properties parse and `outputRunResults`:
- [ ] allow rendering dbt dag at runtime in the Execution topology view if `parse` (new property) is set to `true`
- [ ] add another boolean flag `outputRunResults` indicating whether to output dbt models with custom metadata like duration, which will allow later to expose those as events — ideally, an ION file with one row per dbt model or test (data can be extracted from `run_results.json` generated by dbt after dbt Build/Run completion)

Name | Type | Status | Duration | Completed At
-- | -- | -- | -- | --
model.dwh.stg_customers | Model | success | 0.022 | 2024-03-25T08:46:21.888471Z
model.dwh.stg_orders | Model | success | 0.022 | 2024-03-25T08:46:21.888663Z
model.dwh.stg_locations | Model | success | 0.023 | 2024-03-25T08:46:21.889137Z
model.dwh.stg_order_items | Model | success | 0.023 | 2024-03-25T08:46:21.889313Z
model.dwh.stg_products | Model | success | 0.023 | 2024-03-25T08:46:21.890272Z
model.dwh.stg_supplies | Model | success | 0.023 | 2024-03-25T08:46:21.890450Z
test.dwh.not_null_stg_customers_customer_id | Test | success | 0.042 | 2024-03-25T08:46:21.923868Z
test.dwh.unique_stg_customers_customer_id | Test | success | 0.044 | 2024-03-25T08:46:21.924386Z
test.dwh.not_null_stg_orders_order_id | Test | success | 0.044 | 2024-03-25T08:46:21.935854Z
test.dwh.unique_stg_orders_order_id | Test | success | 0.046 | 2024-03-25T08:46:21.938104Z

## Context

dbt models and tests are difficult to read and inspect in the Gantt view:
![image](https://github.com/kestra-io/plugin-dbt/assets/86264395/396410da-fe3c-47dc-87be-845a18bb601f)

We would like to see those in the Execution Topology view:
![image](https://github.com/kestra-io/plugin-dbt/assets/86264395/b7369ced-f08f-467f-b4fd-b5b3e086d93f)

Simple reproducer to inspect/iterate on the design (you can run it directly with no setup):

```yaml
id: dbtGitDockerDuckDB
namespace: dev

tasks:
- id: dbt
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-demo
branch: main

- id: dbt-build
type: io.kestra.plugin.dbt.cli.Build
debug: true
dockerOptions:
image: ghcr.io/kestra-io/dbt-duckdb:latest
runner: DOCKER
dbtPath: /usr/local/bin/dbt
inputFiles:
.profile/profiles.yml: |
jaffle_shop:
outputs:
dev:
type: duckdb
path: ':memory:'
extensions:
- parquet
target: dev
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start from the dbtRun, dbtBuild, and dbtCll task entry points and use the supplied dbtGitDockerDuckDB reproducer to inspect the current Gantt and Execution Topology behavior. Done means parse can enable runtime dbt DAG rendering and outputRunResults can expose one row per dbt model or test using run_results.json data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.