Pulp offers outdated content

Version
Katello v4.15.0

Describe the bug
A recent host build using Foreman failed the FreeIPA enrollment, digging into why, I found that an apt package listed for update had been removed from the upstream Ubuntu repo. the host I was trying to update could no longer download this package, resulting in the 404 error I was seeing.

The Katello Ubuntu product repo is configured as ‘on demand’. Looking at the logs it's clear that the host tries to download a package no longer available upstream. I tried to sync the repo, but could not get it to update and present the current package version to the host.

To Reproduce
N/A, this was working before.

Expected behavior
Pulp repo to remain in sync with the upstream repo even if using on demand.

Additional context
https://community.theforeman.org/t/katello-deb-repo-out-of-sync-with-upstream/42370

Pulpcore-content logs:

Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb')
Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: pulp [None]: pulpcore.content.handler:WARNING: Could not download remote artifact at 'http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb': 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb'
Feb 20 16:39:34 fm01.some.domain.com pulpcore-content[3050261]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://archive.ubuntu.com/ubuntu/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb')

Start with the on-demand Katello Ubuntu product repository behavior and the pulpcore-content logs showing the 404 for the xxd package URL. Trace how synchronization handles an upstream package that was removed, and consider the issue complete when the repository presents current upstream content without requesting the stale artifact.