NASA-IMPACT / NASA-IMPACT/veda-data
Update Airflow collection pipeline functionality to mimic /api/ingest collection ingest
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Jupyter Notebook
- Stars
- 9
- Forks
- 1
- Avg merge
- 1d 9m
- Merged PRs (30d)
- 1
Description
Issue
Item assets are not created when ingesting through Airflow collection-pipeline (this has been tested in dev).
Current fix
To actually add item_assets, a 2-step process is needed between /api/ingest and airflow dataset pipeline. Where /api/ingest is used to add the collection with the item_assets in the config file, then the proper information will show up in stac collection (see example here). Then after that collection is made, we can use airflow dataset-pipeline to add the assets and render information.
New Feature request
Rather than switching between /api/ingest and airflow, it would be nice to be able to ONLY use airflow to complete this process.
See below for example configs (please note there are different ingest configs for airflow and for /api/ingest. Please let me know if you have any questions. This isn't a high priority, but in the future this can alleviate some headache and create a more unified ingest process.
Config for ingest (this will only work if ingested through /api/ingest and not through airflow collection-pipeline.
{
"id": "test",
"description": "tbd",
"stac_version": "1.0.0",
"links": [
],
"stac_extensions": [],
"title": "test",
"type": "Collection",
"license": "tbd",
"extent": {
"spatial": {
"bbox": [
[
-180,-90,180,90
]
]
},
"temporal": {
"interval": [
[
"2025-07-31T14:45:14.998Z",
"2025-07-31T14:46:14.998Z"
]
]
}
},
"providers": [
{
"name": "tbd",
"description": "tbd",
"roles": [
"tbd"
],
"url": "tbd"
}
],
"dashboard:is_periodic": false,
"dashboard:time_density": "tbd",
"item_assets": {
"DNB_BRDF-Corrected_NTL_Mosaic_SNPP_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "DNB BRDF-Corrected NTL Mosaic SNPP C2",
"description": "Day/Night Band BRDF-Corrected Nighttime Lights Mosaic from Suomi NPP Collection 2"
},
"DNB_BRDF-Corrected_NTL_Mosaic_VJ1_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "DNB BRDF-Corrected NTL Mosaic VJ1 C2",
"description": "Day/Night Band BRDF-Corrected Nighttime Lights Mosaic from NOAA-20 (JPSS-1) Collection 2"
},
"DNB_BRDF-Corrected_NTL_Mosaic_VNP_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "DNB BRDF-Corrected NTL Mosaic VNP C2",
"description": "Day/Night Band BRDF-Corrected Nighttime Lights Mosaic VNP Collection 2"
},
"DNB_BRDF-Corrected_NTLC2_Mosaic_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "DNB BRDF-Corrected NTLC2 Mosaic C2",
"description": "Day/Night Band BRDF-Corrected Nighttime Lights Collection 2 Mosaic"
},
"QF_Cloud_Mask_Mosaic_SNPP_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "QF Cloud Mask Mosaic SNPP C2",
"description": "Quality Flag Cloud Mask Mosaic from Suomi NPP Collection 2"
},
"QF_Cloud_Mask_Mosaic_VJ1_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "QF Cloud Mask Mosaic VJ1 C2",
"description": "Quality Flag Cloud Mask Mosaic from NOAA-20 (JPSS-1) Collection 2"
},
"VNP46A2_QF_Cloud_Mask_Mosaic_VNP_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "QF Cloud Mask Mosaic VNP C2",
"description": "Quality Flag Cloud Mask Mosaic VNP Collection 2"
},
"VNP46A2_QF_Cloud_MaskC2_Cloud_C2_Mosaic_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "QF Cloud MaskC2 Cloud C2 Mosaic C2",
"description": "Quality Flag Cloud Mask Collection 2 Mosaic"
}
}
}
Config that is fed into airflow (slightly different syntax between /api/ingest and airflow for collection ingest)
"license": "CC0-1.0",
"collection": "test",
"title": "test",
"description": "test",
"data_type": "cog",
"spatial_extent": {
"xmin": -180.0,
"ymin": -90.0,
"xmax": 180.0,
"ymax": 90.0
},
"temporal_extent": {
"startdate": "2024-09-18T00:00:00Z",
"enddate": "2024-10-11T23:59:59Z"
},
"stac_version": "1.0.0",
"dashboard:time_density": "day",
"dashboard:is_periodic": false,
"stac_extensions": [
"https://stac-extensions.github.io/render/v1.0.0/schema.json",
"https://stac-extensions.github.io/item-assets/v1.0.0/schema.json"
],
"providers": [
{
"name": "NASA"
}
],
"item_assets": {
"DNB_BRDF-Corrected_NTL_Mosaic_SNPP_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "DNB BRDF-Corrected NTL Mosaic SNPP C2",
"description": "Day/Night Band BRDF-Corrected Nighttime Lights Mosaic from Suomi NPP Collection 2"
},
"DNB_BRDF-Corrected_NTL_Mosaic_VJ1_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "DNB BRDF-Corrected NTL Mosaic VJ1 C2",
"description": "Day/Night Band BRDF-Corrected Nighttime Lights Mosaic from NOAA-20 (JPSS-1) Collection 2"
},
"DNB_BRDF-Corrected_NTL_Mosaic_VNP_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "DNB BRDF-Corrected NTL Mosaic VNP C2",
"description": "Day/Night Band BRDF-Corrected Nighttime Lights Mosaic VNP Collection 2"
},
"DNB_BRDF-Corrected_NTLC2_Mosaic_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "DNB BRDF-Corrected NTLC2 Mosaic C2",
"description": "Day/Night Band BRDF-Corrected Nighttime Lights Collection 2 Mosaic"
},
"QF_Cloud_Mask_Mosaic_SNPP_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "QF Cloud Mask Mosaic SNPP C2",
"description": "Quality Flag Cloud Mask Mosaic from Suomi NPP Collection 2"
},
"QF_Cloud_Mask_Mosaic_VJ1_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "QF Cloud Mask Mosaic VJ1 C2",
"description": "Quality Flag Cloud Mask Mosaic from NOAA-20 (JPSS-1) Collection 2"
},
"VNP46A2_QF_Cloud_Mask_Mosaic_VNP_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "QF Cloud Mask Mosaic VNP C2",
"description": "Quality Flag Cloud Mask Mosaic VNP Collection 2"
},
"VNP46A2_QF_Cloud_MaskC2_Cloud_C2_Mosaic_C2": {
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data",
"layer"
],
"title": "QF Cloud MaskC2 Cloud C2 Mosaic C2",
"description": "Quality Flag Cloud Mask Collection 2 Mosaic"
}
}
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the Airflow collection-pipeline configuration with the /api/ingest collection configuration, focusing on how item_assets and the STAC item-assets extension are handled. Run the existing collection-pipeline flow against the example configuration and verify that one Airflow-only run creates the collection, item assets, and rendered asset information visible in the STAC collection.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100