apache / apache/airflow

cannot use `get_current_context()` in a `@task.virtualenv` task

Open
#34,158 14 comments 1 reaction 0 assignees View on GitHub
area:providers good first issue kind:feature provider:standard
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

On Airflow 2.5.1 (on AWS MWAA), I ran this DAG:

```python
from datetime import datetime, timedelta
from airflow.decorators import dag, task
from airflow.operators.python import get_current_context

@task.virtualenv(system_site_packages=True)
def test():
data_interval_end = get_current_context()['data_interval_end']
print(data_interval_end)

@dag(
start_date=datetime(2023, 9, 6),
schedule="10 * * * *",
)
def bug_test():
test()

the_dag = bug_test()
```

And I got `airflow.exceptions.AirflowException: Current context was requested but no context was found! Are you running within an airflow task?`

I know that I can do it like this:

```python
@task.virtualenv(system_site_packages=True)
def test(data_interval_end=None):
print(data_interval_end)
```

That works fine if I only need the context directly inside that function, but where this actually popped up in practice was a DAG that used some shared lib functions that used `get_current_context`, which of course works fine when called from normal tasks but blew up when called from a virtualenv task.

### What you think should happen instead

ideally, `get_current_context()` should work even if it's called from a virtualenv task.

### How to reproduce

described above

### Operating System

Linux? it's AWS MWAA

### Versions of Apache Airflow Providers

_No response_

### Deployment

Amazon (AWS) MWAA

### Deployment details

_No response_

### Anything else

this was also mentioned in this closed issue: https://github.com/apache/airflow/issues/20974#issuecomment-1674514640

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by running the provided DAG reproduction and read the entry points around get_current_context() and @task.virtualenv. Trace how context is made available across the virtualenv task boundary, including the behavior described in closed issue #20974. Done means shared functions using get_current_context() work inside virtualenv tasks without raising the missing-context exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.