open-telemetry / open-telemetry/opentelemetry-python-contrib
Instrument logs in Celery tasks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 1.1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 16
Description
Before opening a feature request against this repo, consider whether the feature should/could be implemented in the other OpenTelemetry client libraries. If so, please open an issue on opentelemetry-specification first.
Steps to reproduce:
- Instrument logging
- Log outside of celery task
- Log inside of celery task
- Trigger celery task
- Notice that the log from inside the task is not captured
Example:
from fastapi import FastAPI
from celery import Celery
import logging
logging.basicConfig(level=logging.INFO)
app = FastAPI()
celery = Celery('tasks', broker='redis://localhost:6379/0')
@app.get('/')
def root():
#This log is captured
logging.info(f'Begin Task')
task = count_to_10000.delay()
return {'task_id': task.id}
@celery.task
def count_to_10000():
for i in range(1, 10001):
if i % 1000 == 0:
#This log not captured
logging.info(f'Count: {i}')
else:
pass
return 'Counting complete!'
Is your feature request related to a problem?
Yes. Celery tasks can seemingly use the same longer as code outside a celery task. Yet, the logs from within the task do not appear to be captured.
Describe the solution you'd like
The logging instrumentation (or perhaps the celery instrumentation) is updated to somehow capture logs from inside celery tasks.
Describe alternatives you've considered
I have not found alternatives.
Additional context
Datadog seems to have the same issue
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the FastAPI and Celery example with logging outside and inside the task, confirming that only the outer log is captured. Trace the logging instrumentation and Celery task boundary to determine where records are lost. Done means logs emitted inside the Celery task are captured consistently with logs outside it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fastapi, python, redis
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100