Google Vertex AI Feature Store system Dag exceeds import timeout
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
`main` (development)
### What happened and how to reproduce it?
The Google provider's Vertex AI Feature Store system Dag imports `google.cloud.aiplatform_v1beta1` at module scope. Importing that package triggers an expensive third-party import chain while Airflow scans the Dag file.
In a clean Breeze process, importing the unmodified system Dag took 30.505 seconds and loaded `google.cloud.aiplatform_v1beta1`. This exceeds Airflow's default `dagbag_import_timeout` of 30 seconds and can surface as a Dag import timeout or a third-party circular-import failure.
Steps to reproduce:
1. Check out `main` with the Google provider development dependencies installed.
2. Run the Vertex AI Feature Store system Dag import test in Breeze:
```bash
BACKEND=sqlite breeze run pytest airflow-core/tests/unit/always/test_example_dags.py \
-k 'test_should_be_importable and vertex_ai_feature_store' -xvs
```
3. Observe that parsing the Dag loads `google.cloud.aiplatform_v1beta1` at module scope and may exceed the 30-second Dag import timeout.
### What you think should happen instead?
Scanning the system Dag should not load the Vertex AI Feature Store client package. Heavy Google clients and protobuf-related imports should be deferred until task execution so the Dag remains importable within the default timeout.
### Operating System
macOS host with the Apache Airflow Breeze CI container, Python 3.10.
### Deployment
Other Docker-based deployment
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
After removing the parse-time `aiplatform_v1beta1` imports, the system Dag imports without loading that root package and passes Airflow's generic system Dag import, database-query, and connection-access checks.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the Vertex AI Feature Store system DAG and the import behavior described in airflow-core/tests/unit/always/test_example_dags.py, running the provided Breeze pytest command first. Trace the module-scope google.cloud.aiplatform_v1beta1 and related heavy imports, then defer them until task execution. Done means the system DAG imports within the default timeout without loading that root package and passes the stated import, database-query, and connection-access checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100