As a user I can re-upload artifacts if the file has gone missing or corrupted

Open
#1,941 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
28/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
api, backend

Research direction

The issue names artifact upload handling, storage_path, and the existing /repair/ endpoint but no files or tests. Start by tracing those entry points and comparing the missing-file and corrupted-file cases. Agree on the recovery approach and API behavior with maintainers; done means users can restore missing or corrupted artifact data without the current checksum error.

Written by the indexing model from the issue text.

Description

Feature

Author: @ipanova (ipanova@redhat.com)

Redmine Issue: 7791, https://pulp.plan.io/issues/7791

If an artifact has gone missing or corrupted(bit rot) there is no way to re-upload it back to the filesystem

  1. upload an artifact
  2. rm /var/lib/pulp/artifacts/<some_artifact> or corrupt it
  3. upload same artifact
  4. 400 error with {
    "non_field_errors": [
    "sha384 checksum must be unique."
    ]
    }

While we have /repair/ endpoint it will not work for the operations where artifact has no remoteartifact.

Problem statements:

1. If a file is missing it is impossible to upload a new one
  • when saving artifact add try/except, look for existing one
  • verify whether storage_path is an existing location if not update it with the newly uploaded bits
  • Issue 400 due to duplicated artifact, but in addition return the href of the existing artifact.
2. If a file is corrupted it is impossible to re-upload and replace it with a valid one
  • option1 Running repair can find corrupted files. It should remove the corrupted file to get back to the case outlined in 1. .
    • Repair can be run against specific repo version, potentially can extend the functionality to repair a specific artifact/content
  • option2 We could recalculate the checksum on all upload attempts.
    • Might be a lot of overhead for a rare failure
  • option3 Introduce a flag which will be specified at upload time E.g. --repair, or --force, or --validate-checksum.
    It will replace broken bits if checksum of the newly uploaded file matches a checksum in the DB.
    The recalculation of a checksum will happen on_demand this way and not for every upload attempt.
Dominant language
Python
Stars
598
Forks
168
Avg merge
1d 4h
Merged PRs (30d)
86

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pulp/pulpcore

All issues in pulp/pulpcore

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.