Add a rebuild cache

Description of the new feature / enhancement

When rebuilding the entire index, it takes a long time as each manifest must be fully parsed and rebuilt. However, many of these manifests may not have changed since the last time a rebuild was run. With nearly 60,000 manifests, it would be beneficial to have some method of doing a partial rebuild.

Proposed technical implementation details

When a rebuild is performed, a copy of the manifests and the indexes could be saved off to a storage blob as a gzip. When the next rebuild is performed, this gzip could be downloaded and expanded, and the indexes loaded into memory as if it were the publishing pipeline. Then, instead of rebuilding the index from scratch, each manifest could be compared. If the manifest has changed, then update the index based upon the diff from the old manifest file to the new manifest file. If there was no change in the manifest, the index does not need to be updated. Once all the manifests have been processed, the new indexes can be published and a copy of the manifests and indexes can be saved off as the cache for the next rebuild.

Of course the pipelines will still need to have an option to perform a full rebuild, if necessary, but adding a caching layer could significantly reduce the amount of time it takes by starting from the last known-good index.

With this caching strategy, it could also be beneficial to perform a rebuild on a regular cadence (every 3 months?) to help ensure a well-maintained cache.

Start by locating the rebuild pipeline and the manifest and index processing paths described in the issue. Determine how a full rebuild currently loads and publishes manifests, then define how cached state, change detection, and a forced full rebuild would fit those paths. Done means unchanged manifests avoid unnecessary rebuilding while updated manifests produce correct indexes and the cache can be refreshed.