apache / apache/airflow

Add `execute_tasks_new_python_interpreter` argument to `PythonOperator`

Open
#41,093 5 comments 3 reactions 0 assignees View on GitHub
area:core good first issue kind:feature provider:standard
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Currently from my knowledge it is only possible to force all python tasks to execute in a new
interpreter for every execution globally via this setting:

- https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter

Would it be possible to add this argument to the `PythonOperator` so it is possible to control this behavior at the task level?

### Use case/motivation

- https://github.com/apache/airflow/discussions/40956

### Related issues

- https://github.com/apache/airflow/discussions/40956

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the PythonOperator and the existing execute-tasks-new-python-interpreter configuration described in the linked configuration reference. Review the related discussion for the task-level use case. Done means the operator can control whether its Python task runs in a new interpreter, independently of the global setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.