Caching issues again after uploading to app catalog

Target SharePoint environment

SharePoint Online

What SharePoint development model, framework, SDK or API is this about?

SPFx 1.22.1

What browser(s) / client(s) have you tested

• 💥 Microsoft Edge

Describe the bug / error

We're seeing similar caching issues again that were happening a year or two ago and fixed. After uploading to the tenant or site app catalog the new version is showing in the catalog but not being shown on sites that use it for 30+ minutes or sometimes it will show the new version and revert to the old as you click through various pages on the site. This is happening for apps that are 100 KB or 50 MB.

This makes it very difficult to release the code especially if we have APIs that are also released at the same time that do not work with the old version. Web parts are essentially broken until the new version appears on the site. This also makes it hard for our QA team to do smoke tests after releases and creates a lot of downtime that are clients are not expecting.

Let me know if there's any information I can pass along to help debug this.

Previous times this happened
https://github.com/SharePoint/sp-dev-docs/issues/8915
https://github.com/SharePoint/sp-dev-docs/issues/8803

Steps to reproduce

1. Package solution and upload to tenant or site app catalog.
2. Verify new version is uploaded
3. Refresh site that contains web parts from solution.
4. Old web parts are still showing

Expected behavior

New version should appear on sites immediately or within a few minutes.