Data Gitops in the Product Metadata Repo

Sketching out the flow, from our chat with GIS on April 1. Still mentally churning how to make this the easiest for all involved.

Overview

How can we operationalize the product-metadata repo to initiate distribution, in a more gitops-like approach.

We should record the versions (including the current version) in the product metadata repo. PR sketch here.

Flow:

1. GIS user opens a PR with a change to a version (e.g. PLUTO 25v4 -> 26v1). The PR should kick off a dry-run / draft of distribution and post the revision to the Pull Request. Maybe it should also kick off packaging, which would create all assets like pdfs and OTI xlsx's for the modified products/datasets.
2. The PR should kick off a comparison between main and the PR branch to calculate a diff in expected current versions for product.dataset.destinations and posts it to the PR. It also generates all metadata assets (e.g. OTI xlsx's), maybe as a pre-release?
3. They merge the PR to main, which kicks off a diff to determine changed versions.
4. This kicks off distribution.
5. Maybe we run a nightly action to scrape versions from all destinations and compare to our expected version. This is the product-metadata version of nightly QA

E.g. you might open a Pull request to change the current_version in lion/metadata.yml. This calculates a diff for each dataset and destination under LION:

lion.atomic_polygons.socrata:v1 -> lion.atomic_polygons.socrata:v2
lion.rpl.socrata:v1 -> lion.rpl.socrata:v2
...

We kick off a separate distribution job for each of these. And now instead of generating the assets (specifically the OTI xlsx) on the fly, we use what's generated by the release.

This would also make use of the existing override behavior, so if you wanted to pin a version of a dataset or destination, you could do that easily.

Review the product-metadata repo flow, the linked PRs #2304 and #2313, and the existing override behavior, starting with lion/metadata.yml. Define the agreed version-diff, asset-generation, distribution, and nightly-check boundaries; done means the workflow is specified clearly enough to implement and validate.