SPIKE: Investigate separation strategy for Triggerer and DAG Processor for server-client separation
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
Investigate where Triggerer and DAG Processor should live in the AIP-72 server-client separation architecture.
## Background
**Triggerer and DAG Processor:**
- Both are currently in airflow-core
- Both have dependencies on Task SDK execution-time components
- They create circular dependency issues between server and SDK
## Open Questions
### Where should Triggerer & DAG Processor live? Server or Client?
**Option 1: Create `airflow-core-execution` provider**
- Contains Task SDK Execution logic, worker, trigger, and DAG Processor
- **Pros**: Isolated execution concerns, clear separation
- **Cons**: Package explosion, coupling between `apache-airflow-task-sdk` and `apache-airflow-core-execution`
**Option 2: Move Triggerer & DAG Processor to Task SDK**
- Relocate both components to `apache-airflow-task-sdk`
- **Pros**: Execution components together, simpler dependency graph, enables future multi-language DAG authoring
- **Cons**: Task SDK becomes heavier, multi-language implementation complexity
Can we do them in separate phases i.e. not have explicit dependencies in metadata but import them dynamically if they are installed.
**Multi-language SDK Implications:**
- **Future-proofing**: Enables Go SDK and other languages to have native DAG definition & processing
- **Language-specific optimizations**: Each SDK could optimize for their ecosystem (Go's concurrency, etc.)
- **Consistency challenges**: Risk of different behaviors across language implementations
- **Maintenance burden**: Multiple Triggerer/DAG Processor implementations to maintain
Contributor guide
Research direction
Start with the AIP-72 server-client separation architecture described in the issue, then trace Triggerer and DAG Processor in airflow-core and their dependencies on Task SDK execution-time components. Compare the two proposed package boundaries and dynamic-import approach; done means documenting a justified separation strategy and its multi-language SDK implications.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, backend-api-design
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100