Using `start_from_trigger` with `context` and/or `template_fields`
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Discussed in https://github.com/apache/airflow/discussions/41944
Originally posted by **gil-tober** September 2, 2024
Hey all,
We are running Airflow on AWS EKS running `CeleryKubernetesExecutor` and have ~100K daily tasks and ~1500 daily dag_runs.
In Airflow 2.10.0 the `start_from_trigger` feature was introduced for deferrable tasks and it's a feature that could greatly improve our performance and scaling of the cluster.
We have a lot of tasks that are deferrable (mainly Redshift and BigQuery queries).
Looking and the docs for this feature, trying to implement it and looking at the feature code, it seems that
1. `template_fields` are not being rendered before they are passed to the trigger
2. There is no way to pass `context` to the trigger in order to render the fields that we need to render.
Is there any way to achieve this? is there any future plans to implement fields rendering before the trigger is called?
Contributor guide
Research direction
Begin with the linked discussion and the start_from_trigger entry point, comparing the documented behavior with how context and template_fields are currently handled. Before coding, establish the intended rendering semantics for deferrable tasks and a concrete acceptance case for Redshift or BigQuery; done should include agreed behavior and tests demonstrating it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100