pulp / pulp/pulp-operator

Changing content checksums list results in errors and crashloopbackoff

Open
#1,344 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Issue
Dominant language
Go
Stars
88
Forks
66
Avg merge
53m
Merged PRs (30d)
1

Description

Version
Please provide the versions of the pulp-operator and pulp images in use.

$ helm -n pulp list
NAME	NAMESPACE	REVISION	UPDATED                                 	STATUS  	CHART              	APP VERSION 
pulp	pulp     	1       	2024-09-03 16:54:04.919341315 +0300 EEST	deployed	pulp-operator-0.1.0	1.0.1-beta.4

apiVersion: repo-manager.pulpproject.org/v1beta2

kind: Pulp

metadata:
  name: pulp
  namespace: pulp

spec:
  allowed_content_checksums:
    - sha1
    - sha256
    - sha512

  object_storage_s3_secret: s3-secret

  database:
    # when changing pulp-postgresql env values, update secret suffix here
    external_db_secret: external-pulp-postgresql-secret-74844b4846

  cache:
    enabled: true
    external_cache_secret: external-pulp-redis-secret-d8d2dhd8bk

  api:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        cpu: 1
        memory: 512Mi

  content:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

  worker:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 500m
        memory: 500Mi
      limits:
        cpu: 2
        memory: 1Gi

Describe the bug
Added section to config:

allowed_content_checksums:
    - sha1
    - sha256
    - sha512

Applied changes to Pulp deployment and all api, content and workers are crashing.

$ kubectl -n pulp logs pod/pulp-api-5bb4d46d59-7fcgx 
Waiting on postgresql to start...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.

Expected behavior
Pulp would work and accept listed content checksums.

Additional context
Job to update content checksums is failing

$ kubectl -n pulp logs pod/pulp-update-content-checksums-sxkkw-h86qt 
Waiting on postgresql to start...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Checking for database migrations
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/__init__.py", line 34, in <module>
    from .exporter import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
    from pulpcore.app.models.repository import Repository
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 34, in <module>
    from pulpcore.cache import Cache
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
    from .cache import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 17, in <module>
    from pulpcore.app.settings import settings
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Database migrated!
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 179, in handle
    for chunk in fp.chunks(CHUNK_SIZE):
  File "/usr/local/lib/python3.9/site-packages/django/core/files/base.py", line 55, in chunks
    self.seek(0)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/utils.py", line 45, in <lambda>
    seek = property(lambda self: self.file.seek)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 48, in _get_file
    self._file = self.storage.open(self.name, "rb")
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 22, in open
    return self._open(name, mode)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 488, in _open
    f = S3File(name, mode, self)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 133, in __init__
    self.obj.load(**params)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 565, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Pulp manifest's allowed_content_checksums setting and the pulpcore-manager handle-artifact-checksums entry point. Reproduce the change with helm and inspect the kubectl logs from the API, worker, content, and update-content-checksums pods. Done means the checksum update job completes and the Pulp pods stop crash-looping while accepting the listed checksums.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes, python
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.