pulp / pulp/pulp_deb

'NoneType' object has no attribute 'artifact_set_sha256' when syncing a recently updated deb remote

Open
#719 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Triage-Needed
Dominant language
Python
Stars
71
Forks
91
Avg merge
1d 19h
Merged PRs (30d)
22

Description

Version

Installed and setup via ansible (pulp_installer)

    {
      "component": "core",
      "version": "3.21.3",
      "package": "pulpcore"
    },
    {
      "component": "deb",
      "version": "2.20.0",
      "package": "pulp_deb"
    }

Describe the bug

It looks like there is an issue comparing the indices for a remote when syncing a repository after the remote has updated. When running a sync job, it returns with this error:

.Error: Task /pulp/api/v3/tasks/28fcc35c-15cf-4e00-be84-98041c6dd7f2/ failed: ''NoneType' object has no attribute 'artifact_set_sha256''

The stack trace for the error is:

File "/pulp/pulpenv/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
  result = func(*args, **kwargs)
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 170, in synchronize
  DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
  loop.run_until_complete(pipeline)
File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
  return future.result()
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
  await asyncio.gather(*futures)
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
  await self.run()
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 544, in run
  await asyncio.gather(
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 679, in _handle_distribution
  await asyncio.gather(*sub_tasks)
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 763, in _handle_component
  await asyncio.gather(*pending_tasks)
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 861, in _handle_package_index
  if previous_package_index.artifact_set_sha256 == package_index.artifact_set_sha256:

To Reproduce

It may be difficult to reproduce. This is only happening on the microsoft repositories. We are syncing arch amd64 for jammy, focal, and bionic. In order to reproduce it you need to have a previous version of the repository/release. If you setup a brand new remote and repository, it doesn't encounter this issue when syncing, since it has nothing to compare from.

These are the basic settings

pulp deb remote create --name "microsoft_deb_focal" --architecture amd64 --distribution focal --url "https://packages.microsoft.com/ubuntu/20.04/prod"
pulp deb repository create --description "Microsoft focal repo" --remote microsoft_deb_focal --name repo_microsoft_deb_focal
pulp deb repository sync --name repo_microsoft_deb_focal

Expected behavior

The repository should sync and pull in the new updates without any issues.

Additional context

The remote microsoft repos we're mirror'ing were updated several days ago. The day they were updated is when our sync's started to fail. It's only for those repos. I am uncertain if they configured something incorrectly in their release, or it's simply a bug with the plugin. The timing is odd nonetheless. These are the URLs:

if it's of any relevance, here is the remote config for one of the above. They're identical aside from name and URL

{
  "pulp_href": "/pulp/api/v3/remotes/deb/apt/819cf3bf-4559-4027-8ed6-fa2c27eb462b/",
  "pulp_created": "2022-12-09T22:11:46.085397Z",
  "name": "deb_remote_microsoft_focal",
  "url": "https://packages.microsoft.com/ubuntu/20.04/prod",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": null,
  "pulp_labels": {},
  "pulp_last_updated": "2023-02-06T14:53:20.757223Z",
  "download_concurrency": 2,
  "max_retries": 10,
  "policy": "on_demand",
  "total_timeout": 1200.0,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": 60.0,
  "headers": null,
  "rate_limit": null,
  "distributions": "focal",
  "components": null,
  "architectures": "amd64",
  "sync_sources": false,
  "sync_udebs": false,
  "sync_installer": false,
  "gpgkey": null,
  "ignore_missing_package_indices": false
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pulp_deb/app/tasks/synchronizing.py at _handle_package_index, around line 861, and trace how the previous package index is selected during synchronization. Reproduce with an existing Microsoft Ubuntu repository and an updated remote, then verify that synchronization completes and pulls the new updates instead of failing on the missing previous index.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.