Transfer reprocessed HLS environmental justice granules and full provider metadata to production

What

We need to transfer a custom subset of HLS granules from two collections in the staging version of veda-backend to production. These collections include detailed provider metadata and have complex storage paths in S3 that require this work to be done by hand (the flat, minimal, structure of the datasets pipelines used for many veda collections would cause us to lose the detailed structure of these granules and metadata which are required for multi band maps and analyses like those used in this notebook: https://nasa-impact.github.io/veda-docs/notebooks/quickstarts/hls-visualization.html)

Background: These collections include a small number of tiles that were selected to spotlight before and after imagery from two environmental justtice stories and demonstrate how titiler+pgstac can be used to investigate that data in the cloud. Early HLS (harmonized landsat sentinel) data were selected for reprocessing (CRS data were missing from these older files). The data were downloaded from the host DAAC for reprocessing so we need to copy this set of reprocessed files to veda-data-store and then we need to update the stac metadata to href the new veda-data-store location (much more detailed than our in-house dataset metadata so we do not want to execute the build-stac pipeline for these).

Historical VEDA HLS issues

• https://github.com/NASA-IMPACT/veda-data-pipelines/issues/147
• https://github.com/NASA-IMPACT/veda-data-pipelines/issues/89

1. Transfer the files

• Copy the complete contents of the HLS landsat reprocessed granules from s3://covid-eo-data/hlsl30-ej-reprocessed/ to s3://veda-data-store/hlsl30-ej-reprocessed/ without modifying the detailed key paths. I.e. s3://covid-eo-data/hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017157T144341.v2.0/HLS.L30.T19QHA.2017157T144341.v2.0.B01.tif will go to s3://veda-data-store/hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017157T144341.v2.0/HLS.L30.T19QHA.2017157T144341.v2.0.B01.tif
• Do the same for the Sentinel reprocessed granules from s3://covid-eo-data/hlss30-ej-reprocessed to s3://veda-data-store/hlss30-ej-reprocessed

2. Update the hrefs in stac item records with a python script or notebook that can be reused if needed

• Select all of the stac items from the hlsl30-ej-reprocessed collection and iterate over the assets to update the hrefs (now covid-eo-data will be replaced with veda-data-store will be replaced in every asset href). Publish these updated items. Alternatively, the objects copied in the first step include stac metadata that would be nice to update and use in this process (for example s3://covid-eo-data/hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017193T150719.v2.0/HLS.S30.T19QHA.2017193T150719.v2.0_stac-ej-reprocessed.json is a complete stac item record describing all band assets and thumbnails but the asset hrefs are covid-eo-data).
• Do the same for sentinel series hlss30-ej-reprocessed

3. Cleanup reprocessed environmental justice story config in veda-data

• Ensure that collection metadata is correct for hlsl30-ej-reprocessed and hlss30-ej-reprocessed (should be already)
• Remove any airflow/DAG configuration for these collections to prevent future confusion
• Copy an artifact of how the metadata were updated to veda-data//transformation-scripts to make the href update process reproducible
• Process to walk HLS buckets in veda-data store and select appropriate stac-metadata files (there are multiple versions) and post those items to stac-api/ingestions endpoint