apache / apache/airflow

Websocket trigger for long-lived deferable operators

Open
#21,139 5 comments 1 reaction 0 assignees View on GitHub
area:async-operators kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Create a trigger for deferrable operators that uses a websocket client to send a long-lived/long-running task request to an external remote websocket server that handles (or intiates the handling of) the requested task.
The Async trigger then triggers when the remote server replies
The remote websocket server will be out of the scope of airflow and not implemented here.

### Use case/motivation

A common use case is to run long lived tasks asynchronously when the duration of execution is unknown.
Http requests are not the best for long lived connections so a http trigger may not be optimal.
The currently provided timer trigger does not meet this requirement
A websocket request would be better suited for this (I would even go further and propose a messaging system(e.g kafka) is the best option in such a case - I might open a different feature request if this current feature is approved)

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; start by reviewing existing deferrable operators and the timer trigger to understand the relevant entry points. Define the websocket request and reply contract, and consider the work complete when the trigger can maintain the long-lived interaction and resume when the remote server replies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.