apache / apache/airflow

Google GKE system Dags use deprecated cluster fields

Open
#69,825 1 comment 0 reactions 1 assignee Claimed by @viiccwen View on GitHub
area:providers kind:bug needs-triage provider:google
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

`main` (development)

### What happened and how to reproduce it?

This is a follow-up to #69822.

Removing the Vertex AI Feature Store system Dag's expensive parse-time imports allowed the generic system Dag import and serialization checks to proceed far enough to expose a separate problem in five Google Kubernetes Engine system Dags.

The GKE examples set `initial_node_count` directly on the cluster payload. The Google provider warns that this cluster-level field is deprecated and recommends configuring the initial node count on a node pool instead. Consequently, checks that treat provider deprecation warnings as errors fail while importing these example Dags.

Steps to reproduce:

1. Check out `main` with the Google provider development dependencies installed.
2. Run the affected example Dag import tests in Breeze while treating the warning as an error:

```bash
breeze run pytest airflow-core/tests/unit/always/test_example_dags.py \
-k 'example_kubernetes_engine' \
-W 'error:The body field.*is deprecated:UserWarning' \
-q --disable-warnings
```

3. Observe that importing the GKE system Dags fails because their cluster payloads use the
deprecated top-level `initial_node_count` field.

The affected examples are:

- `example_kubernetes_engine.py`
- `example_kubernetes_engine_async.py`
- `example_kubernetes_engine_job.py`
- `example_kubernetes_engine_kueue.py`
- `example_kubernetes_engine_resource.py`

### What you think should happen instead?

The GKE system Dags should define an explicit node pool and set `initial_node_count` on that node pool. They should import and serialize without emitting the cluster-level field deprecation warning.

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.