otel_metrics task is too expensive — high DB CPU usage and long execution time
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 598
- Forks
- 168
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 86
Description
The otel_metrics scheduled task (pulpcore.app.tasks.telemetry.otel_metrics) runs every 5 minutes and currently takes approximately 1 minute and 30 seconds to complete. During execution, the underlying query consumes ~3 vCPUs from the database.
Here is where we believe the issue is happening (pulpcore/app/tasks/telemetry.py:31-33):
space_usage_per_domain = Artifact.objects.values("pulp_domain__name").annotate(
total_size=Sum("size", default=0)
)
- A telemetry task that is meant to be lightweight is placing significant load on the database.
- 3 vCPU consumption for a single periodic query reduces capacity available for actual content operations (sync, publish, etc.).
- The task takes 1m30s out of every 5-minute cycle (30% duty cycle), meaning a worker is occupied with telemetry for a disproportionate amount of time.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting pulpcore/app/tasks/telemetry.py:31-33 and the otel_metrics scheduled task, then measure the Artifact aggregation query against the reported database load and runtime. Done means the telemetry task no longer takes about 1 minute 30 seconds or consumes roughly 3 database vCPUs during its five-minute cycle.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, observability, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100