apache / apache/airflow

Airflow can not open path in zipped DAG folder

Open
#59,310 13 comments 0 reactions 0 assignees View on GitHub
affected_version:3.1 area:core kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

3.1.4

### If "Other Airflow 2/3 version" selected, which one?

_No response_

### What happened?

The bug in the following issue was never solved:

https://github.com/apache/airflow/issues/10160

Jinja2 templating does not work and one can not change the template_searchpath such that the (for example) sql file is recognized.

There is just a workaround using the open_maybe_zipped utils function and using it to open the file manually before giving the path to the Operator, like this:

```
from airflow.utils.file import open_maybe_zipped

open_maybe_zipped("/home/airflow/gcs/dags/test.zip/test_sql/test.sql").read()`
```

### What you think should happen instead?

_No response_

### How to reproduce

In `/home/airflow/gcs/dags/test.zip/test_sql/test.sql`:
```
SELECT column_a FROM test
```

In `/home/airflow/gcs/dags/test.zip/test.py`:

```
from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from airflow.utils.file import open_maybe_zipped

# Define DAG
with DAG(
dag_id="test_import_path_dag",
schedule_interval=None,
default_args = {"retries": 0}
) as dag:

test = BigQueryInsertJobOperator(
task_id="test_task",
configuration={
"query": {
"query": "test_sql/test.sql" , # PATH HERE IS NOT FOUND WHEN DAGS ARE ZIPPED
"useLegacySql": False,
"destinationTable": {
"projectId": "test",
"datasetId": "test",
"tableId": "test",
},
}
},
)

test
```

### Operating System

Linux

### Versions of Apache Airflow Providers

_No response_

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with Airflow's open_maybe_zipped utility and the BigQueryInsertJobOperator example using /home/airflow/gcs/dags/test.zip/test.py and test_sql/test.sql. Compare how the zipped DAG path is handled; done means the SQL path is recognized and Jinja2 templating works without manually opening the file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.