open-telemetry / open-telemetry/opentelemetry-python-contrib

Fix and re-enable the flaky celery tests

Open
#653 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug good first issue help wanted
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

Some celery tests were disabled a long time ago because of inconsistency.

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/tests/opentelemetry-docker-tests/tests/celery/test_celery_functional.py#L36
https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/tests/opentelemetry-docker-tests/tests/celery/test_celery_functional.py#L146
https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/tests/opentelemetry-docker-tests/tests/celery/test_celery_functional.py#L193
https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/tests/opentelemetry-docker-tests/tests/celery/test_celery_functional.py#L209
https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/tests/opentelemetry-docker-tests/tests/celery/test_celery_functional.py#L479

All of the above tests either use .delay() or .apply_async() to trigger the Celery tasks. This means a span is generated for the producer, then a message is put on a queue/broker. A celery worker then picks up the message and executes the relavant task which generates the consumer span. This is tricky as there can be some delay between producing and consuming. It looks like our tests do not take this delay into account and expect both producer and consumer spans to be available immediately after calling delay/apply_async which obviously fails a lot of times.

Solution here would be to wait till the expected number of spans arrive to memory exporter or wait on a condition fulfilled by the task to be executed. Whichever solution we pick should have a timeout with a assertion error so tests don't get stuck forever in case the message never arrives.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/opentelemetry-docker-tests/tests/celery/test_celery_functional.py and inspect the disabled tests at the referenced lines, along with how the memory exporter observes spans. Run the Celery functional test file to reproduce the timing failures. Done means the tests are re-enabled, wait for producer and consumer spans with a timeout, and fail with an assertion instead of hanging when a task does not arrive.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.