Hanging test_send_tasks_to_celery_hang in quarantine
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Body
It looks like the `test_send_tasks_to_celery_hang` is actually hanging occasionally in the ways that it has not been caught by our test timeouts.
Test timeouts work in the way, that they register for SIGALRM and handle them, however it seems that this test - especially in Amazon MWAA team environment does not timeout and hangs forever. This is likely because of the signal handling / resetting that is implemented by celery libraries and our tests.
This test has been added in order to test some nasty race condition with multiprocessing https://github.com/apache/airflow/pull/15989 but likely there is another related race condition that triggers it occassionally.
After the move of celery executor (and celery executor tests) to provider, this test was executed as part of "Providers" tests and this move caused the test to hang pretty deterministically in AWS MWWA team environment. For now we are quarantining the test to allow the tests to pass for MWAA team, but also this test is a strong suspect for generating mysterious "test failed" conditions in CI of Airflow - if such test would hang, Github Actions is known to loose logs for jobs that have been stuck in the way that they have to be terminated forecefully. There is a big chance this test is causing it.
But it is extremely difficult to reproduce in Airflow CI in a repeatable way.
We should investigate the test before 2.7.0 gets released. The good news is that if we have a hypothesis that we would like to test, we can simply unquarantine the test with the fix and ask the MWAA team (@vincbeck @ferruzzi @o-nikolas were involved) to run the PR in their environment - which should rather quickly give answer if the problem has been fixed.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start with the quarantined test_send_tasks_to_celery_hang in the provider's Celery executor tests, then inspect the SIGALRM timeout and reset behavior used by the test and Celery libraries. Reproduce the hang, ideally in the MWAA environment, and confirm that the test no longer hangs and can be removed from quarantine.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100