[CT-3225] [Feature] permit python models to declare `ref` and `source` models independently of data-access
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this your first time submitting a feature request?
- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
Permit python models to declare upstream refs/sources separately from accessing the data in those upstream models.
Maybe like this:
```python
@ref('model_A')
def model(dbt, session):
pass
```
Or this:
```python
def model(dbt, session):
dbt.declare_ref('model_A')
pass
```
Python models currently rely on `dbt.ref` and `dbt.source` to implicitly declare dependence on other models. This pattern follows the use of the `ref` and `source` macros in `.sql` models insofar as the declaration coincides with data access.
This is really cool!
However, these functions are not the only way a user might access an upstream model's data using python, and depending on circumstance, may not be preferred (or even possible).
### Describe alternatives you've considered
#### 1. Don't do this
The main alternative to consider seems to be: don't do this.
Reading through the [original proposal](https://github.com/dbt-labs/dbt-core/discussions/5261) for python models, I gather that this functionality may have been deliberately omitted.
I think I understand the core motivation (maybe not!): python models, by definition, represent a significant break from strictly ELT workflows, but the spirit of ELT might be preserved so long as computation remains relatively remote from whatever client is issuing commands.
It's possible that making the lineage graph accessible outside of the context of a full blown adapter (where compute is sufficiently scalable to handle warehouse-sized workloads) encourages non-idiomatic, local usage and the better alternative is to leave things the way they are.
However, I can, today, nonetheless, create python models which ignore the `dbt` object entirely and they work. I get my materialization. Out of the box. I'm just penalized insofar as that materialization has no access to the core utility of dbt, and I'm forced to hack it back into the lineage graph by declaring the resulting table as a source. Because this remains possible -- the penalty feels more like a nuisance.
Not to mention popular adapters like dbt-duckdb which put the lie a little bit to the notion of remote computation being essential.
#### 2. `dbt.ref` could accept a custom handler
If the desire is to continue to couple the declaration of an upstream relationship with data access, the same end could be accomplished with something like:
```python
def custom_handler(*args, **kwargs):
return warehouse.to_dataframe("select * from model_A")
def model(dbt, session):
df = dbt.ref('model_A', handler=custom_handler)
```
There may be other motivations for this kind of pattern, but it seems heavy-handed as a solution to the current issue.
#### 3. Supporting an optional `config` object, and supporting ref/source declarations there
To increase parity with sql models, instead of merely using `model`, dbt could look for an optional variable named `config`:
```python
config = {
'materialized': 'table',
'refs': ['model_A']
}
def model(dbt, session):
pass
```
This seems like a fine thing, but, similar to the custom handler idea, feels like a bigger design choice than is justified by the limited scope of this issue.
### Who will this benefit?
All users who implement or want to implement "plain old" python models, without using the `dbt` object to load data. Currently these users lose the magic of the `ref` function and all that comes with it. Their lineage graph/dag is broken.
### Are you interested in contributing this feature?
Happy to, if the preferred implementation can be briefly sketched out.
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.