dbt-labs / dbt-labs/dbt

[Feature] `flags.INVOCATION_COMMAND` for programmatic dbt invocations

Open
#10,313 5 comments 0 reactions 0 assignees View on GitHub
engine:v1 help_wanted type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

When programmatically invoking dbt in a separate python script (i.e. a Pytest function), the CLI args passed to that external python script are in `sys.argv`, which causes `flags.INVOCATION_COMMAND` to be equal to `dbt {sys.argv}`, even when the arguments may have originally been passed to the external Python script.

For instance, when logging the `flags.INVOCATION_COMMAND` from a model during programmatic invocation from a Pytest function, I see

```
INVOCATION_COMMAND="dbt tests/my_test.py -sxv"
```

### Expected Behavior

In reality, the programmatic invocation was more like

```
dbt_runner.invoke(["compile", "--empty", "--select", "my_empty_test_model"], project_dir=..., profiles_dir=...)
```

I would have expected `INVOCATION_COMMAND` to be `"dbt compile --empty --select my_empty_test_model"`, but instead it was `"dbt tests/my_test.py -sxv"` (my arguments passed to `pytest`).

### Steps To Reproduce

1. Initialize an empty dbt project
2. Inside one of the default models, add a Jinja block for logging the flags variable:
```
{% do log("INVOCATION_COMMAND = " ~ flags.INVOCATION_COMMAND, info=true) %}
```
4. Create a python script in the root project folder, called `main.py`
5. Fill that main.py script with these contents:

```py
from dbt.cli.main import dbtRunner
from pathlib import Path

DBT_PROJECT_DIR = Path(__file__).parent

def cli():
dbt_runner = dbtRunner()
dbt_runner.invoke(["compile"], project_dir=DBT_PROJECT_DIR, profiles_dir=DBT_PROJECT_DIR)

if __name__ == "__main__":
cli()
```

6. Call the python script directly, passing any additional arguments you want:

```
python main.py hello world
```

The results show that the INVOCATION_COMMAND is not "dbt compile" like we'd expect, but "dbt hello world."

### Relevant log output

```shell
14:06:00 Running with dbt=1.8.2
14:06:01 Registered adapter: snowflake=1.8.3
14:06:01 Found 2 models, 4 data tests, 444 macros
14:06:01
14:06:02 Concurrency: 4 threads (target='dev')
14:06:02
14:06:02 INVOCATION_COMMAND = dbt hello world
```

### Environment

```markdown
- OS: Mac OSX Ventura 13.6
- Python: 3.11.2 (via pyenv 2.3.15)
- dbt:
- dbt-core: 1.8.2
- dbt-snowflake: 1.8.3
```

### Which database adapter are you using with dbt?

snowflake

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.