pulp / pulp/pulpcore

otel_metrics task is too expensive — high DB CPU usage and long execution time

Open
#7,475 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Task
Dominant language
Python
Stars
598
Forks
168
Avg merge
1d 4h
Merged PRs (30d)
86

Description

The otel_metrics scheduled task (pulpcore.app.tasks.telemetry.otel_metrics) runs every 5 minutes and currently takes approximately 1 minute and 30 seconds to complete. During execution, the underlying query consumes ~3 vCPUs from the database.

Here is where we believe the issue is happening (pulpcore/app/tasks/telemetry.py:31-33):

  space_usage_per_domain = Artifact.objects.values("pulp_domain__name").annotate(
      total_size=Sum("size", default=0)
  )
  • A telemetry task that is meant to be lightweight is placing significant load on the database.
  • 3 vCPU consumption for a single periodic query reduces capacity available for actual content operations (sync, publish, etc.).
  • The task takes 1m30s out of every 5-minute cycle (30% duty cycle), meaning a worker is occupied with telemetry for a disproportionate amount of time.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting pulpcore/app/tasks/telemetry.py:31-33 and the otel_metrics scheduled task, then measure the Artifact aggregation query against the reported database load and runtime. Done means the telemetry task no longer takes about 1 minute 30 seconds or consumes roughly 3 database vCPUs during its five-minute cycle.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases, observability, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.