apache / apache/airflow

Client-Server Separation: DAG Processor & Triggerer

Open
#61,802 1 comment 0 reactions 0 assignees View on GitHub
area:core area:Triggerer kind:feature kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Body

Epic for addressing client-server separation in DAG Processor and Triggerer components.

### Why

Both DAG Processor and Triggerer run user code and they should stay in task sdk for that reason. They currently use an in process execution API but they should shift towards communicating with API server via API for DB needs.

Related AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services

### Scope

- **DAG Processor**: Eliminate task SDK dependencies on airflow core for DAG parsing and serialization. Consider moving it to task SDK.
- **Triggerer**: Eliminate task SDK dependencies on airflow core for trigger definition and execution.

### Committer

- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.