apache / apache/airflow

Issues upgrading from Airflow 3.3.0 to 3.3.1

Open
#71,772 10 comments 0 reactions 0 assignees View on GitHub
kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.3.1

### What happened and how to reproduce it?

We are currently running Airflow 3.3.0 in Kubernetes using the Airflow Helm Chart v1.22 for deployment. All Dags are healthy. The Dags are fetched via Git Dag Bundle.

After deploying, the dag processor messages errors like `Processor for DagFileInfo(rel_path=PosixPath(''), bundle_name='', bundle_path=PosixPath('/tmp/airflow/dag_bundles//tracking_repo', bundle_version=None) with PID NNN has been running for 50.01 seconds, exceeding the timeout of 50.00 seconds. Killing it! [airflow.dag_processing.manager.DagFileProcessorManager] loc=manager.py:1621` and the overview table now shows "# Errors" > 0

We have a very basic "airflow_health_check" Dag to verify the Airflow workers Dag runtime:

```py
from airflow import XComArg
from airflow.sdk import dag, task
from airflow.sdk.exceptions import AirflowFailException
from datetime import datetime, timedelta

HEALTH_CHECK_MESSAGE: str = "Health check successful"
HEALTHY: str = "healthy"
UNHEALTHY: str = "unhealthy"

@dag(
schedule="*/2 * * * *",
start_date=datetime(2026, 7, 1),
dagrun_timeout=timedelta(minutes=3),
is_paused_upon_creation=False,
catchup=False,
)
def airflow_health_check() -> None:
@task.bash
def health_check(text: str) -> str:
return f"echo '{text}'"

@task
def result_is_healthy(*, result: XComArg, check_against: str) -> bool:
return str(result) == check_against

@task
def print_result(is_healthy: bool) -> None:
print("is_healthy: " + str(type(is_healthy)) + " " + str(is_healthy))
print(HEALTHY if is_healthy == True else UNHEALTHY)
if not is_healthy == True:
raise AirflowFailException()

health_check_result: XComArg = health_check(HEALTH_CHECK_MESSAGE)
result_is_healthy_bool: bool = result_is_healthy(
result=health_check_result, check_against=HEALTH_CHECK_MESSAGE
)
print_result(result_is_healthy_bool)

airflow_health_check()
```

With Airflow 3.3.0, the Dag Runs are green.
Shortly after the upgrade to 3.3.1, the Dag Runs are stuck in "queued", but the Dag itself is still active.
With the Dag processor catching up and now throwing errors, the Dag gets "Deactivated".

### What you think should happen instead?

It should just work? Idk, do I overlook something?

### Operating System

Container image apache/airflow:3.3.1-python3.13

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

amazon, celery, cncf-kubernetes, common-compat, common-io, common-messaging, common-sql, databricks, docker, elasticsearch, fab, ftp, git, google, grpc, hashicorp, http, keycloak, microsoft-azure, mysql, odbc, openlineage, opensearch, postgres, redis, sendgrid, sftp, slack, smtp, snowflake, ssh, standard

### Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.34.0
apache-airflow-providers-celery==3.23.1
apache-airflow-providers-cncf-kubernetes==10.21.0
apache-airflow-providers-common-compat==1.18.0
apache-airflow-providers-common-io==1.8.0
apache-airflow-providers-common-messaging==2.0.4
apache-airflow-providers-common-sql==2.1.0
apache-airflow-providers-databricks==7.18.1
apache-airflow-providers-docker==4.5.9
apache-airflow-providers-elasticsearch==6.9.0
apache-airflow-providers-fab==3.8.0
apache-airflow-providers-ftp==3.15.2
apache-airflow-providers-git==0.4.2
apache-airflow-providers-google==22.3.0
apache-airflow-providers-grpc==3.9.5
apache-airflow-providers-hashicorp==4.8.0
apache-airflow-providers-http==6.0.5
apache-airflow-providers-keycloak==0.8.2
apache-airflow-providers-microsoft-azure==14.1.0
apache-airflow-providers-mysql==6.6.1
apache-airflow-providers-odbc==4.12.3
apache-airflow-providers-openlineage==2.20.0
apache-airflow-providers-opensearch==1.12.0
apache-airflow-providers-postgres==7.0.1
apache-airflow-providers-redis==4.5.0
apache-airflow-providers-sendgrid==4.2.4
apache-airflow-providers-sftp==6.0.1
apache-airflow-providers-slack==9.10.2
apache-airflow-providers-smtp==3.0.3
apache-airflow-providers-snowflake==6.16.0
apache-airflow-providers-ssh==6.0.1
apache-airflow-providers-standard==1.17.0

### Official Helm Chart version

1.22.0 (latest released)

### Kubernetes Version

v1.35.1

### Helm Chart configuration

Incomplete, only an excerpt. Please let me know if there might be something other relevant to share.

```yaml
images:
airflow:
repository: private.registry/airflow
tag: snapshot-#commit-sha # based on apache/airflow:3.3.1-python3.13
airflowVersion: 3.3.1
config:
azure_remote_logging:
remote_wasb_log_container: airflow-remote-logs
celery_broker_transport_options:
visibility_timeout: 14400
core:
auth_manager: airflow.providers.keycloak.auth_manager.keycloak_auth_manager.KeycloakAuthManager
test_connection: Hidden
keycloak_auth_manager:
realm: application
requests_pool_size: 20
server_url:
logging:
delete_local_logs: "False"
remote_base_log_folder: wasb://logs
remote_log_conn_id: airflow-remote-logging-wasb
remote_logging: "True"

dagProcessor:
dagBundleConfigList:
- classpath: airflow.providers.git.bundles.git.GitDagBundle
kwargs:
git_conn_id:
refresh_interval: 30
tracking_ref: main
name:
```

### Docker Image customizations

Dockerfile:
```dockerfile
ARG AIRFLOW_CONTAINER_VERSION=3.3.1-python3.13

FROM apache/airflow:${AIRFLOW_CONTAINER_VERSION}

USER root

COPY org_ca.pem /tmp/org_ca.pem
RUN csplit -z -f /usr/local/share/ca-certificates/org_ca_ \
--suffix-format='%03d.crt' \
/tmp/org_ca.pem \
'/-----BEGIN CERTIFICATE-----/' '{*}' \
&& update-ca-certificates \
&& rm /tmp/org_ca.pem

USER airflow

COPY upstream-requirements.txt upstream-requirements.txt
RUN --mount=type=secret,id=pip-conf,dst=/etc/pip.conf,mode=0440 \
pip install --no-cache-dir \
--constraint /home/airflow/constraints.txt \
--requirement upstream-requirements.txt

# install custom providers and packages in separate pip install command to avoid dependency version clash with constraints
COPY custom-requirements.txt custom-requirements.txt
RUN --mount=type=secret,id=pip-conf,dst=/etc/pip.conf,mode=0440 \
pip install --no-cache-dir \
--requirement custom-requirements.txt

ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
```

upstream-requirements.txt:
```
# Environmernt variable ${AIRFLOW_VERSION} comes from the upstream Airflow image and is resolved by pip
apache-airflow[cloudpickle,databricks,keycloak,microsoft.azure,otel]==${AIRFLOW_VERSION}

# Airflow components not covered via extras, versions are covered by constraints file
virtualenv
```

custom-requirements.txt:
```
# Environment variable ${AIRFLOW_VERSION} comes from the upstream Airflow image and is resolved by pip
# Pin Airflow version to prevent pip from changing it during dependency resolution
apache-airflow==${AIRFLOW_VERSION}

```

### Anything else?

To be honest, I am a bit lost where to look at else. Please let me know :)

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start at airflow.dag_processing.manager.py:1621 and inspect the GitDagBundle configuration and DAG processor timeout behavior during the 3.3.0-to-3.3.1 upgrade. Reproduce with the supplied airflow_health_check DAG and the Airflow Helm Chart configuration, comparing processor errors and queued runs between versions. Done means the root cause is identified and DAG processing and scheduling work normally after the upgrade.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, git, helm, kubernetes, python
Domain
data-engineering, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.