Add deferrable mode to SFTPOperator
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
The SFTPOperator currently runs in synchronous mode only, blocking
a worker slot for the entire duration of every file transfer.
For large file transfers, this wastes worker resources unnecessarily.
Proposed Solution
Add deferrable=True parameter to SFTPOperator following the same
pattern used in other Airflow providers. When deferrable=True,
the operator will defer execution to a trigger, freeing the worker
slot immediately and allowing the Triggerer to poll for completion.
Implementation Plan
- Add SFTPOperatorTrigger class in triggers/sftp.py
- Add deferrable param to SFTPOperator.__init__()
- Add defer() call in execute() when deferrable=True
- Add execute_complete() callback method
- Add unit tests
Related
Similar to deferrable mode already implemented in SFTPSensor.
I am willing to submit a PR for this.
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.