apache / apache/airflow

SPIKE: Investigate separation strategy for Triggerer and DAG Processor for server-client separation

Open
#51,552 1 comment 0 reactions 0 assignees View on GitHub
area:task-execution-interface-aip72 area:Triggerer
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

Investigate where Triggerer and DAG Processor should live in the AIP-72 server-client separation architecture.

## Background
**Triggerer and DAG Processor:**
- Both are currently in airflow-core
- Both have dependencies on Task SDK execution-time components
- They create circular dependency issues between server and SDK

## Open Questions

### Where should Triggerer & DAG Processor live? Server or Client?

**Option 1: Create `airflow-core-execution` provider**
- Contains Task SDK Execution logic, worker, trigger, and DAG Processor
- **Pros**: Isolated execution concerns, clear separation
- **Cons**: Package explosion, coupling between `apache-airflow-task-sdk` and `apache-airflow-core-execution`

**Option 2: Move Triggerer & DAG Processor to Task SDK**
- Relocate both components to `apache-airflow-task-sdk`
- **Pros**: Execution components together, simpler dependency graph, enables future multi-language DAG authoring
- **Cons**: Task SDK becomes heavier, multi-language implementation complexity

Can we do them in separate phases i.e. not have explicit dependencies in metadata but import them dynamically if they are installed.

**Multi-language SDK Implications:**
- **Future-proofing**: Enables Go SDK and other languages to have native DAG definition & processing
- **Language-specific optimizations**: Each SDK could optimize for their ecosystem (Go's concurrency, etc.)
- **Consistency challenges**: Risk of different behaviors across language implementations
- **Maintenance burden**: Multiple Triggerer/DAG Processor implementations to maintain

Contributor guide

Open the contributing guide

Research direction

Start with the AIP-72 server-client separation architecture described in the issue, then trace Triggerer and DAG Processor in airflow-core and their dependencies on Task SDK execution-time components. Compare the two proposed package boundaries and dynamic-import approach; done means documenting a justified separation strategy and its multi-language SDK implications.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, backend-api-design
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.