Dill fails to serialize context in Python 3.11 correctly
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Body
There is a regression that needs investigation. One of our tests is failing only on Python 3.11
`tests/operators/test_python.py::TestPythonVirtualenvOperator::test_airflow_context`
This test is part of PlainAssert suite and it has been skipped so far but after https://github.com/apache/airflow/pull/35160 the PlainAssert (having only that test) have been brought back to regular tests and it turned out that it fails for Python 3.11
The error is about dill serializing the context:
```
INFO airflow.utils.process_utils:process_utils.py:186 Output:
INFO airflow.utils.process_utils:process_utils.py:190 Traceback (most recent call last):
INFO airflow.utils.process_utils:process_utils.py:190 File "/tmp/venv-call7xpd4uip/script.py", line 17, in
INFO airflow.utils.process_utils:process_utils.py:190 arg_dict = dill.load(file)
INFO airflow.utils.process_utils:process_utils.py:190 ^^^^^^^^^^^^^^^
INFO airflow.utils.process_utils:process_utils.py:190 File "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 270, in load
INFO airflow.utils.process_utils:process_utils.py:190 return Unpickler(file, ignore=ignore, **kwds).load()
INFO airflow.utils.process_utils:process_utils.py:190 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO airflow.utils.process_utils:process_utils.py:190 File "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 472, in load
INFO airflow.utils.process_utils:process_utils.py:190 obj = StockUnpickler.load(self)
INFO airflow.utils.process_utils:process_utils.py:190 ^^^^^^^^^^^^^^^^^^^^^^^^^
INFO airflow.utils.process_utils:process_utils.py:190 TypeError: code() argument 13 must be str, not int
ERROR airflow.models.taskinstance.TaskInstance:taskinstance.py:2612 Task failed with exception
```
And likely connected with similar issues reported by others.
The way how to reproduce it:
1) Run `breeze --python 3.11`
2) In the test remove the `or PY311` in
```
@pytest.mark.skipif(
os.environ.get("PYTEST_PLAIN_ASSERTS") != "true" or PY311,
reason="assertion rewriting breaks this test because dill will try to serialize "
...
)
def test_airflow_context(self):
```
3) Run this command:
```
PYTEST_PLAIN_ASSERTS="true" pytest tests/operators/test_python.py::TestPythonVirtualenvOperator::test_airflow_context --assert=plain
```
(Note - for this test assert rewriting in Pytest must be disabled that's why we have the ENV variable and --assert=plain`.
Example failure: https://github.com/apache/airflow/actions/runs/6710410150/job/18236357337 (and you can see that Python 3.8 - 3.10 are all green, only Python 3.11 is affected).
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start with tests/operators/test_python.py::TestPythonVirtualenvOperator::test_airflow_context and reproduce under Python 3.11 using the provided breeze and pytest commands, with the PY311 skip removed. Trace the dill serialization failure shown in the test output and compare behavior with Python 3.8–3.10. Done means the context test passes on Python 3.11 without retaining the skip.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100