apache / apache/airflow

Google Vertex AI Feature Store system Dag exceeds import timeout

Open Beginner friendly
#69,822 0 comments 0 reactions 0 assignees View on GitHub
area:providers kind:bug needs-triage provider:google
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

`main` (development)

### What happened and how to reproduce it?

The Google provider's Vertex AI Feature Store system Dag imports `google.cloud.aiplatform_v1beta1` at module scope. Importing that package triggers an expensive third-party import chain while Airflow scans the Dag file.

In a clean Breeze process, importing the unmodified system Dag took 30.505 seconds and loaded `google.cloud.aiplatform_v1beta1`. This exceeds Airflow's default `dagbag_import_timeout` of 30 seconds and can surface as a Dag import timeout or a third-party circular-import failure.

Steps to reproduce:

1. Check out `main` with the Google provider development dependencies installed.
2. Run the Vertex AI Feature Store system Dag import test in Breeze:

```bash
BACKEND=sqlite breeze run pytest airflow-core/tests/unit/always/test_example_dags.py \
-k 'test_should_be_importable and vertex_ai_feature_store' -xvs
```

3. Observe that parsing the Dag loads `google.cloud.aiplatform_v1beta1` at module scope and may exceed the 30-second Dag import timeout.

### What you think should happen instead?

Scanning the system Dag should not load the Vertex AI Feature Store client package. Heavy Google clients and protobuf-related imports should be deferred until task execution so the Dag remains importable within the default timeout.

### Operating System

macOS host with the Apache Airflow Breeze CI container, Python 3.10.

### Deployment

Other Docker-based deployment

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

After removing the parse-time `aiplatform_v1beta1` imports, the system Dag imports without loading that root package and passes Airflow's generic system Dag import, database-query, and connection-access checks.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the Vertex AI Feature Store system DAG and the import behavior described in airflow-core/tests/unit/always/test_example_dags.py, running the provided Breeze pytest command first. Trace the module-scope google.cloud.aiplatform_v1beta1 and related heavy imports, then defer them until task execution. Done means the system DAG imports within the default timeout without loading that root package and passes the stated import, database-query, and connection-access checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.