Websocket trigger for long-lived deferable operators
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
Create a trigger for deferrable operators that uses a websocket client to send a long-lived/long-running task request to an external remote websocket server that handles (or intiates the handling of) the requested task.
The Async trigger then triggers when the remote server replies
The remote websocket server will be out of the scope of airflow and not implemented here.
### Use case/motivation
A common use case is to run long lived tasks asynchronously when the duration of execution is unknown.
Http requests are not the best for long lived connections so a http trigger may not be optimal.
The currently provided timer trigger does not meet this requirement
A websocket request would be better suited for this (I would even go further and propose a messaging system(e.g kafka) is the best option in such a case - I might open a different feature request if this current feature is approved)
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
The issue names no files or tests; start by reviewing existing deferrable operators and the timer trigger to understand the relevant entry points. Define the websocket request and reply contract, and consider the work complete when the trigger can maintain the long-lived interaction and resume when the remote server replies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100