NASA-IMPACT / NASA-IMPACT/veda-features-api-cdk

Migrate data from old veda-features-api stack

Open
#9 6 comments 1 reaction 1 assignee View on GitHub

@anayeaye is already working on this.

Since Mar 18, 2025.

Dominant language
Python
Stars
0
Forks
2
Avg merge
8d 5h
Merged PRs (30d)
1

Description

Description

We need to migrate the legacy features API (https://firenrt.delta-backend.com/) which is hosted in UAH to the production features API which is deployed to: https://openveda.cloud/api/features/

Periodic ingests in production will be happen using the SM2A automated vector DAG which will run on chron.

This stack is the one we are providing ongoing support to and will eventually deprecateveda-features-api.

Steps:

  • tear down existing features API in MCP
  • update production sm2a to veda-data-airflow release >= v0.13.6-dev.3
  • create snapshot of veda-wfs3-west2-staging RDS in UAH
  • encrypt and share snapshot to MCP
  • deploy veda-features-api in MCP using snapshot id
  • create and upload automation json in tf shared state bucket in MCP for each collection. See example from staging:
{
    "bucket": "veda-data-store-staging",
    "discovery": "s3",
    "collection": "eis-fedsoutput-lfarchive",
    "filename_regex": "^(.*)lf_(perimeter|fireline|newfirepix)_nrt_(conus|borealna|russiaeast).gpkg$",
    "prefix": "EIS/FEDSoutput/LFArchive/",
    "vector": true,
    "id_template": "{}",
    "schedule": "*/15 * * * *",
    "dag": "veda_ingest_vector"
  }
  • update sm2a deploy secret TF_VAR_vector_secret_name to configure the new features service ingest and cycle change (re-deploy)

Tuning and acceptance testing moved to NASA-IMPACT/veda-architecture#600

  • openveda.cloud/api/features <> firenrt.delta-backend.com parity testing and documentation
  • enable cloudfront caching for api/features
  • enable cloudfront invalidation in sm2a vector ingest by updating the SM2A secret and the scheduled configs
  • update docs.openveda.cloud API links and notebook

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.