pulp / pulp/pulpcore

Have an alternative way to check if the pulp-worker is alive

Open
#6,615 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature
Dominant language
Python
Stars
598
Forks
168
Avg merge
1d 4h
Merged PRs (30d)
86

Description

Is your feature request related to a problem? Please describe.
Currently, to check if a worker is online you need to send a request to the API.
When working on a Kubernetes or Openshift environment, it's not possible to configure
the readinessProbe or livenessProbe to request the API to verify if a worker is alive
or not. Also, when scaling up the pulp-worker to hundreds of replicas it could impact
the API and its operation.
Pulp itself offers an endpoint for API and Content health checking, but not for pulp-worker.

Describe the solution you'd like
Having a way to check if a worker is alive, directly from it's container.
Maybe pulp-worker could offer a channel using some unix socket file, where using socat
or nc we could ask the worker if it's considered online or not. If the application
takes too much time, we could consider it offline and restart its pod/container.

Describe alternatives you've considered
Any file with any data that could be consumed by a shell utility.

Additional context
Now, we're using the wait_on_postgres.py script to check if the pod is able to access
the database, and assume that the pulp-worker is online, which can be false sometimes.
Also, when scaling up the number of workers, this causes an overhead of database connections
around 10% of the number of workers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with wait_on_postgres.py and the worker's existing online-state handling; the issue does not name a worker source entry point. Compare a container-local health signal with Kubernetes readiness and liveness probe needs. Done means a probe can determine whether the pulp-worker is online without issuing API requests or adding the described database-connection overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.