Add capability to exit tasks from trigger
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Description
We should be able to exit the "task" from the trigger.
We can already log from triggers so that's one hurdle out of the way.
The remaining hurdle I think is xcom, given that triggers run in asyncio.
One solution that comes to mind is add a queue listener running in a thread that processes the “messages” and handles submitting to xcom.
Another would be use sqlalchemy's asyncio support.
Sqlalchemy asyncio support would not help us though with custom xcom backend -- so maybe we either do just listener or we add an "optimized" path when not using custom xcom backend.
One implementation question is... would you just make it so that the trigger must call T`rigger.xcom_push` manually (which sends to this queue listener) or would you also want to make it possible, perhaps in an analogy with execute , possible to send the xcom payload(s) in the trigger event, or perhaps through a dedicated `Trigger.success` method?
I think that I’d probably prefer forcing Trigger writers to do the xcom push manually because … an interface to handle both the default return_value as well as handling multiple key-values isn’t quickly coming to mind.
cc: @jedcunningham @phanikumv @vandonr-amz
### Use case/motivation
This would save users money or increase cluster bandwidth because for sensor-like deferrable tasks, no task needs to get spun up just to exit.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.