kestra-io / kestra-io/plugin-dbt
Adding custom outputFiles to dbtCLI task causes dbt artifacts to no longer be rendered
- Dominant language
- Java
- Stars
- 14
- Forks
- 14
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 10
Description
### Issue description
This issue is not a blocker but sth worth investigating long term.
As long as you add outputFiles property to the dbtCLI task, the models are no longer rendered. We should investigate why
Reproducer:
setup flow:
```yaml
id: upload_dbt_project
namespace: company.datateam.dbt
tasks:
- id: wdir
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: git_clone
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-example
branch: master
- id: upload
type: io.kestra.plugin.core.namespace.UploadFiles
namespace: "{{ flow.namespace }}"
files:
- "glob:**/dbt/**"
```
then:
```yaml
id: dbt_build
namespace: company.datateam.dbt
tasks:
- id: dbt
type: io.kestra.plugin.dbt.cli.DbtCLI
namespaceFiles:
enabled: true
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
projectDir: dbt
commands:
- dbt build --project-dir dbt --profiles-dir dbt
loadManifest:
key: manifest.json
namespace: "{{ flow.namespace }}"
storeManifest:
key: manifest.json
namespace: "{{ flow.namespace }}"
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
```
the above flow will work fine, but then try adding:
```yaml
outputFiles:
- "**/*.json"
```
dbt outputs are no longer rendered
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the provided flow with and without the DbtCLI outputFiles setting, comparing whether dbt artifacts are rendered. Trace the DbtCLI entry point and its outputFiles and artifact-rendering handling; done means custom outputFiles can be enabled without preventing dbt artifacts from rendering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cli, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100