Add extended black marble nightlights data to the API (high-level steps)

For each dataset, we will follow the following steps:

Identify the dataset and what the processing needs are

1. Identify dataset and where it will be accessed from. Check it's a good source with science team. Ask about specific variables and required spatial and temporal extent. Note most datasets will require back processing (e.g. generating cloud-optimized data for historical data).

Data has been uploaded by Ranjay to s3://covid-eo-blackmarble/FinalBMHD_DashboardEvolution.tar.gz. We need to extract the TIFs and move them into s3://climatedashboard-data (either to /delivery for staging for a workflow or if the workflow can go directly from the .tar.gz to unzip into s3://climatedashboard-data/bmhd_30m_monthly_bkp/ we should do that.

I am checking with Ranjay that these data are essentially the same as the original set of nightlights data.

In email I asked:

At a high level, I think I'm just trying to answer "should we ingest these files into the current collection or create a new collection"? And "should the existing data be overridden?"
I think this dataset will include an expanded set of cities, should any of the existing data be replaced?Is there any difference in the bands and spatial and temporal nature of these files from the originals and how the files were produced from the original dataset?

Design the metadata and publish to the Dev API

1. Review conventions for generating STAC collection and item metadata:

   • Collections: https://github.com/NASA-IMPACT/delta-backend/issues/29 and STAC version 1.0 specification for collections
   • Items: https://github.com/NASA-IMPACT/delta-backend/issues/28 and STAC version 1.0 specification for items
   • NOTE: The delta-backend instructions are specific to datasets for the climate dashboard, however not all datasets are going to be a part of the visual layers for the dashboard so I believe you can ignore the instructions that are specific to "dashboard" extension, "item_assets" in the collection and "cog_default" asset type in the item

2. After reviewing the STAC documentation for collections and items and reviewing existing scripts for generating collection metadata (generally with SQL) and item metadata, generate or reuse scripts for your collection and a few items to publish to the testing API. There is some documentation and examples for how to generate a pipeline or otherwise document your dataset workflow in https://github.com/NASA-IMPACT/cloud-optimized-data-pipelines. We would like to maintain the scripts folks are using to publish datasets in that repo so we can easily re-run those datasets ingest and publish workflows if necessary.

3. If necessary, request access and credentials to the dev database and ingest and publish to the Dev API. Submit a PR with the manual or CDK scripts used to run the workflow to publish to the Dev API and include links to the published datasets in the Dev API

Publish to the Staging API

Once the PR is approved, we can merge and publish those datasets to the Staging API