Nested plugins with the same name causes import error
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Apache Airflow version
2.5.3
### What happened
## Overview
When a Python file in a submodule in the plugins folder has the same name as a Python file in the root of the plugins folder and another file in the submodule tries to import from the Python file in the root of the plugins folder, the dag that uses that submodule folder will have an import error.
Additionally, there is some different behavior (more explained in the "How to reproduce" section) when that Python file with the same name is specifically called `utils.py`.
## Screenshots
In the screenshots, I replaced the full path with the identifier: `$AIRFLOW_HOME` since my local filepath isn't important to the issue.
### Airflow 2.5.3
This is with the filename as `helpers.py`

This is with the filename as `utils.py`

### Airflow 2.3.4
In previous versions of Airflow (specifically 2.3.4), the dag still ran as expected.
This is with the filename as `helpers.py`. The dag ran as expected.

This is with the filename as `utils.py`. The dag ran as expected, but there is an error message in the UI still.

## Thoughts
I think the reason why name the filename as `utils.py` might cause a problem is because there is a module in the Airflow Python package called "utils" as well - although, I have not quite looked into this yet.
### What you think should happen instead
The dag should run as expected no matter what the filenames are in the plugins folder. And you should be able to have a filename in a submodule that is the same as a filename in the root of the plugins folder without running into errors.
### How to reproduce
## Setup
Create these files and folder structure.
### Folder structure:
```
dags/
-> hello_world.py
plugins/
-> fruits/
-> __init__.py
-> apple.py
-> helpers.py
-> helpers.py
```
### Contents of Files
```python
# dags/hello_world.py
from airflow.models import DAG
from airflow.operators.python import PythonOperator
from fruits.apple import apple
from datetime import datetime
with DAG(
dag_id="hello_world",
description="Hello world",
start_date=datetime(2023, 4, 1),
schedule_interval=None,
catchup=False
) as dag:
PythonOperator(
task_id="greeting",
python_callable=apple
)
```
```python
# plugins/fruits/apple.py
from helpers import greeting
def apple():
return greeting() + " apple"
```
```python
# plugins/fruits/helpers.py
def stuff():
return "asdf"
```
```python
# plugins/helpers.py
def greeting():
return "hi"
```
### Initialization Commands
Run these commands to initialize airflow
```bash
airflow db init
airflow users create --role Admin --username admin --email admin@local.com --firstname admin --lastname admin --password admin
```
I also set `load_examples = False` in the `airflow.cfg` file.
## Testing
### Airflow 2.5.3
Upon opening the webserver (after running `airflow webserver` and `airflow scheduler`), you are greeted with this error:
```python
Broken DAG: [$AIRFLOW_HOME/dags/hello_world.py] Traceback (most recent call last):
File "$AIRFLOW_HOME/dags/hello_world.py", line 4, in
from fruits.apple import apple
File "$AIRFLOW_HOME/plugins/fruits/apple.py", line 1, in
from helpers import greeting
ImportError: cannot import name 'greeting' from 'helpers' ($AIRFLOW_HOME/plugins/fruits/helpers.py)
```
And there will be no dags listed in the UI. However, if you run the `airflow dags list` command, it will no through an import error. Here is the output of that command:
```
dag_id | filepath | owner | paused
============+================+=========+=======
hello_world | hello_world.py | airflow | False
```
Similarly, the `airflow dags list-import-errors` command returns this:
```
No data found
```
If I make the dag fail by giving it a bad import (say `from fruits2.apple import apple` in `dags/hello_world.py`, the expected result shows the traceback.
Interestingly, if I change both `helpers.py` files to `utils.py`, then I also get this error message in the Airflow webserver UI
```python
Broken plugin: [$AIRFLOW_HOME/plugins/fruits/apple.py] cannot import name 'greeting' from 'utils' ($AIRFLOW_HOME/plugins/fruits/utils.py)
```
### Airflow 2.3.4
When rerunning this test on Airflow 2.3.4, the dag was able to run properly. When the filename is `helpers.py`, there are no error messages. However, when the filename is `utils.py`, this error shows up in the UI even though the dag can run properly:
```python
Broken plugin: [$AIRFLOW_HOME/plugins/fruits/apple.py] cannot import name 'greeting' from 'utils' ($AIRFLOW_HOME/plugins/fruits/utils.py)
```
And in both cases, the `airflow dags list` and `airflow dags list-import-errors` commands return as if there are no errors.
### Operating System
Ubuntu 20.04 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql==1.3.4
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-http==4.2.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-sqlite==3.3.1
### Deployment
Virtualenv installation
### Deployment details
Actually, just a conda environment - Python 3.8.12. Airflow package installed with pip.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Reproduce the import conflict using dags/hello_world.py, plugins/fruits/apple.py, plugins/fruits/helpers.py, and plugins/helpers.py, then repeat with utils.py on Airflow 2.5.3. Compare behavior with Airflow 2.3.4 and inspect plugin and DAG loading during the reproduction. Done means nested plugins can reuse filenames without import errors, while the DAG and plugin import-error commands report consistent results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100