apache / apache/airflow

Add deferrable mode to SFTPOperator

Open
#65,475 2 comments 0 reactions 0 assignees View on GitHub
area:providers good first issue kind:feature provider:sftp
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

The SFTPOperator currently runs in synchronous mode only, blocking
a worker slot for the entire duration of every file transfer.
For large file transfers, this wastes worker resources unnecessarily.

Proposed Solution
Add deferrable=True parameter to SFTPOperator following the same
pattern used in other Airflow providers. When deferrable=True,
the operator will defer execution to a trigger, freeing the worker
slot immediately and allowing the Triggerer to poll for completion.

Implementation Plan
- Add SFTPOperatorTrigger class in triggers/sftp.py
- Add deferrable param to SFTPOperator.__init__()
- Add defer() call in execute() when deferrable=True
- Add execute_complete() callback method
- Add unit tests

Related
Similar to deferrable mode already implemented in SFTPSensor.

I am willing to submit a PR for this.

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.