apache / apache/airflow

Allow jinja templating connection ids for all third party operators

Open
#35,259 8 comments 0 reactions 0 assignees Claimed by @shahar1 View on GitHub
good first issue kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

We use private staging and prod S3s(Ceph clusters for example) in our office.
So there are often cases where DAGs are running with only connection ids changed.
We prefer to use Param rather than to use hardcoded connection ids to make our code reusable.
I only gave an example for Amazon operator, but templating connection ids is required for other operators too.

Why is it needed?
Code reusability

### Use case/motivation

```Python3
with DAG(
dag_id="example_s3",
params={
"aws_conn_id": Param("", type="string"),
},
...
) as dag:

create_object = S3CreateObjectOperator(
task_id="create_object",
s3_bucket=bucket_name,
s3_key=key,
data=DATA,
replace=True,
aws_conn_id="{{ params.aws_conn_id }}", # Params enable us to provide runtime configuration
)
```

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the S3CreateObjectOperator example and its aws_conn_id parameter, then trace how third-party operators handle connection IDs and DAG params. Done means a templated connection ID works consistently across the requested third-party operators, including the S3 use case described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.