Missing files in on-demand mode

Version
"versions": [
{
"component": "core",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulpcore.app",
"domain_compatible": true
},
{
"component": "deb",
"version": "3.4.0",
"package": "pulp_deb",
"module": "pulp_deb.app",
"domain_compatible": false
},
{
"component": "ostree",
"version": "2.4.4",
"package": "pulp-ostree",
"module": "pulp_ostree.app",
"domain_compatible": true
},
{
"component": "certguard",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulp_certguard.app",
"domain_compatible": true
},
{
"component": "file",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulp_file.app",
"domain_compatible": true
},
{
"component": "container",
"version": "2.22.0",
"package": "pulp-container",
"module": "pulp_container.app",
"domain_compatible": false
},
{
"component": "ansible",
"version": "0.23.1",
"package": "pulp-ansible",
"module": "pulp_ansible.app",
"domain_compatible": false
},
{
"component": "python",
"version": "3.12.5",
"package": "pulp-python",
"module": "pulp_python.app",
"domain_compatible": true
},
{
"component": "maven",
"version": "0.8.1",
"package": "pulp-maven",
"module": "pulp_maven.app",
"domain_compatible": false
},
{
"component": "rpm",
"version": "3.27.2",
"package": "pulp-rpm",
"module": "pulp_rpm.app",
"domain_compatible": true
}
],

Describe the bug
We have been using Pulp in on-demand mode for the Ubuntu archives (noble, noble-updates, noble-security) and only for the amd64 architecture. After several months of use, we observed that some artifacts previously used, reports 404 even though the files are present in /var/lib/pulp.

We haven't performed any resynchronizations other than the initial sync. For all repositories (noble, noble-updates, noble-security), we have created their own remote.

Pulp has been deployed in our Kubernetes cluster using the Pulp Operator. After the first occurence of the missing files we performed a redeploy.

Additional context
Here is the script I've used to test artifact presence:

#!/bin/bash
REPO_VERSION_HREF="/pulp/api/v3/repositories/deb/apt/0193d9f0-2e4d-7a4d-8d04-69a7855f2d11/versions/1/"
OUTPUT_FILE="artifacts-noble-20250107.txt"
BASE_URL="https://pulp.cloud.prz/pulp/content/ubuntu-noble-amd64/"
echo "" > $OUTPUT_FILE

# Step 1: List all content units in the repository version
CONTENT_UNITS=$(pulp deb content list --repository-version "$REPO_VERSION_HREF" --limit 10000)

# Step 2: Check each artifact's details
echo "Downloaded artifacts for repository version $REPO_VERSION_HREF:"
echo "$CONTENT_UNITS" | jq -c '.[]' | while read -r content_unit; do
    ARTIFACT_HREF=$(echo "$content_unit" | jq -r '.artifact // empty')
    if [ -n "$ARTIFACT_HREF" ]; then
       REL_PATH=$(echo "$content_unit" | jq -r '.relative_path // empty')
       FULL_URL=$BASE_URL/$REL_PATH
       if curl --head --silent --fail "$FULL_URL" > /dev/null; then
         echo "Exists: $FULL_URL" | tee -a "$OUTPUT_FILE"
       else
         echo "Not Found: $FULL_URL" | tee -a "$OUTPUT_FILE"
       fi
    fi
done

Here is the script for the initial sync

#!/bin/bash -x
REPOSITORY_NAME='ubuntu-noble-amd64'
REMOTE_NAME="${REPOSITORY_NAME}"
DISTRIBUTION_NAME="${REPOSITORY_NAME}"
SIGNING_SERVICE="deb-signing-service"
REMOTE_OPTIONS=(
  --url=http://archive.ubuntu.com/ubuntu/
  --distribution=noble
  --component=main
  --component=multiverse
  --component=universe
  --component=restricted
  --architecture=amd64
  --policy on_demand
)
pulp deb remote create --name=${REMOTE_NAME} ${REMOTE_OPTIONS[@]}
pulp deb repository create --name=${REPOSITORY_NAME} --remote=${REMOTE_NAME}
pulp deb repository sync --name=${REPOSITORY_NAME}
pulp deb publication create --repository=${REPOSITORY_NAME}
pulp deb distribution create --name=${DISTRIBUTION_NAME} --base-path=${REPOSITORY_NAME} --repository=${REPOSITORY_NAME}

pulp deb repository sync --name=${REPOSITORY_NAME} --remote=${REMOTE_NAME}
PUB_HREF=$(pulp deb publication create --repository=${REPOSITORY_NAME} | jq -r '.pulp_href')
pulp deb distribution update --name=${REPOSITORY_NAME} --publication=${PUB_HREF}  --base-path=${REPOSITORY_NAME}

Start with the artifact-presence script and the initial-sync commands in the issue, reproducing the Ubuntu amd64 on-demand setup and the reported 404s after several months. Trace how the deb remote, repository version, publication, and distribution serve artifacts that are present in /var/lib/pulp. Done means previously fetched files remain retrievable from the distribution and a regression test covers the case.