Airflow can not open path in zipped DAG folder
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
3.1.4
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
The bug in the following issue was never solved:
https://github.com/apache/airflow/issues/10160
Jinja2 templating does not work and one can not change the template_searchpath such that the (for example) sql file is recognized.
There is just a workaround using the open_maybe_zipped utils function and using it to open the file manually before giving the path to the Operator, like this:
```
from airflow.utils.file import open_maybe_zipped
open_maybe_zipped("/home/airflow/gcs/dags/test.zip/test_sql/test.sql").read()`
```
### What you think should happen instead?
_No response_
### How to reproduce
In `/home/airflow/gcs/dags/test.zip/test_sql/test.sql`:
```
SELECT column_a FROM test
```
In `/home/airflow/gcs/dags/test.zip/test.py`:
```
from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from airflow.utils.file import open_maybe_zipped
# Define DAG
with DAG(
dag_id="test_import_path_dag",
schedule_interval=None,
default_args = {"retries": 0}
) as dag:
test = BigQueryInsertJobOperator(
task_id="test_task",
configuration={
"query": {
"query": "test_sql/test.sql" , # PATH HERE IS NOT FOUND WHEN DAGS ARE ZIPPED
"useLegacySql": False,
"destinationTable": {
"projectId": "test",
"datasetId": "test",
"tableId": "test",
},
}
},
)
test
```
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with Airflow's open_maybe_zipped utility and the BigQueryInsertJobOperator example using /home/airflow/gcs/dags/test.zip/test.py and test_sql/test.sql. Compare how the zipped DAG path is handled; done means the SQL path is recognized and Jinja2 templating works without manually opening the file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100