kestra-io / kestra-io/plugin-dbt

Adding custom outputFiles to dbtCLI task causes dbt artifacts to no longer be rendered

Open
#157 2 comments 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
14
Forks
14
Avg merge
1d 10h
Merged PRs (30d)
10

Description

### Issue description

This issue is not a blocker but sth worth investigating long term.

As long as you add outputFiles property to the dbtCLI task, the models are no longer rendered. We should investigate why

Reproducer:

setup flow:
```yaml
id: upload_dbt_project
namespace: company.datateam.dbt

tasks:
- id: wdir
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: git_clone
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-example
branch: master

- id: upload
type: io.kestra.plugin.core.namespace.UploadFiles
namespace: "{{ flow.namespace }}"
files:
- "glob:**/dbt/**"
```
then:
```yaml
id: dbt_build
namespace: company.datateam.dbt

tasks:
- id: dbt
type: io.kestra.plugin.dbt.cli.DbtCLI
namespaceFiles:
enabled: true
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
projectDir: dbt
commands:
- dbt build --project-dir dbt --profiles-dir dbt
loadManifest:
key: manifest.json
namespace: "{{ flow.namespace }}"
storeManifest:
key: manifest.json
namespace: "{{ flow.namespace }}"
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
```

the above flow will work fine, but then try adding:

```yaml
outputFiles:
- "**/*.json"
```

dbt outputs are no longer rendered

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the provided flow with and without the DbtCLI outputFiles setting, comparing whether dbt artifacts are rendered. Trace the DbtCLI entry point and its outputFiles and artifact-rendering handling; done means custom outputFiles can be enabled without preventing dbt artifacts from rendering.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cli, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.