[DISCUSSION] Backend decisions/architecture guidelines

Background

Since this is a bit of a green-field project I wanted to jot down some notes about how to structure the project.

Some guiding principles:

• Should re-use existing code libraries as much as possible (pip install > forking > re-writing from scratch)
• Should be easy to deploy as a single unit
• Should be easy to re-use constituent parts in another project

Three constituent parts are:

• A tiling api (TiTiler)
• A STAC api (stac-fastapi)
• An ingestion pipeline (Custom implementation)

Everything after this line is up for discussion and debate.

Tiling API:

Option 1: TiTiler as an external dependency

TiTiler is available as a pip-installable dependency. To add the TiTiler routes to an existing FastAPI application:

from titiler.core.factory import TilerFactory

# Create a FastAPI application
app = FastAPI(
    description="A lightweight Cloud Optimized GeoTIFF tile server",
)

# Create a set of COG endpoints
cog = TilerFactory()

# Register the COG endpoints to the application
app.include_router(cog.router, tags=["Cloud Optimized GeoTIFF"])

This will include the following routes: https://developmentseed.org/titiler/endpoints/cog/

This makes the Tiling functionality very easy and straightforward to add as an external dependency to the project - which reduces the amount of code we have write and maintain and ensures that we are able to easily stay up to date with developments to the tiler.

Option 2: Fork TiTiler

I'm not sure why we would want to do this, but I'm including as an option worthy of discussion

STAC API:

Some unknowns about stac-fastapi:

• Is there an out-of-the-box integration with TiTiler somehow?
  • If not, we should fork the stac-fastapi repo, add TiTiler as a pip dependency and add the /cog routes using the method defined above
• How do we deploy stac-fastapi using CDK? I see 2 principle options (which will be discussed further down)
  1. (I will describe this option in further detail than the other, because I'm more familiar it, but by no means does this mean I think its the better of the two options). Add a lambda handler file, which imports the FastAPI app and wraps it with Mangum. Extend the Dockerfile included in the stac-fastapi repo to copy the handler file to the Docker execution context. Deploy the stac-fastapi app as a lambda function with an API Gateway integration, and an RDS instance to host the postgres DB
  2. Deploy the Docker image included in stac-fastapi as an ECS task with an Elastic Load Balancer to provide a public interface to the ECS service.
  • Do we add a Fargate layer for orchestration? (If so, how do Fargate and ECS interact)
  • ELB vs ALB? Is there a difference? Is one better than the other?
  • The RDS instance will likely be deployed within a VPC - how can we ensure the ECS service is able to access the RDS instance inside the VPC?
  • What kind of features does ELB/ALB have that we can take advantage of? Logging? Rate limiting? Role-based authentication?
  • For reference the csdap-orders STAC API is deployed as an ECS service with an ELB

pg_stac:

stac-fastapi is built on top of pg_stac. pg_stac provides the postgres schemas and functions necessary for implementation of the STAC API.

The pg_stac schemas/functions can be applied to the database using 2 options:

• sqitch
• sqlalchemy alembic

I believe the preferred method to apply the schemas/functions is using alembic, which can be run within a python script, using the sqlalchemy psycopg2 database connector.

The csdap-orders STAC API CDK deployment code has an example of defining a custom resource within the CDK stack, where the custom resource is the actual execution of a lambda function, in this case, the lambda function runs the database setup (or "bootstrapping") using alembic. I'm a little bit fuzzy on the details, but I believe this is the overall idea.

Unknowns:

• pg_stac comes with a python utility (pypgstac) for loading data and migrating the database from one version of postgres to the next. Does this utility also contain functionality to "bootstrap" the database?
• Is there an example/implementation somewhere of "bootstrapping" a pg_stac database?

TiTiler-pgstac:

The titiler-pgstac is an extension for TiTiler that connects directly to a pg_stac database to generate mosaics dynamically from STAC queries. This may be an interesting options to explore.

Unknowns:

• Does this extension provide a STAC API endpoint on the tiler? Or just the ability to generate mosaic's from STAC queries?

Due to STAC's widespread adoption and strong specification, we want the tiler to work with any STAC API/database. This means that we would like the tiler to be able to generate mosaics from a STAC API endpoint, rather than directly accessing the database. If the TiTiler-pgstac extension provides a STAC API endpoint, it would be easy to configure the tiler to access the STAC records through any API endpoint, enabling this backend for both scientists that want to build and manage their own STAC API or scientists that already have a STAC API up and running.

If the Titiler-pgstac does not provide a STAC API endpoint then we should stay away from this implementation in order to enable the tiler to be compatible with any STAC API.

API Gateway + Lambda vs ELB + ECS:

Paraphrasing Drew: we used to prefer Lambda for the near infinite scalability, and for the lack of costs when there's no traffic to the application. Some drawbacks to Lambda include limited execution time and memory, inability to make use of multi-threading or to re-use database connections to speed up operations (since each execution may or may not take place in a new container instance). An example of this: GDAL natively implements some cacheing mechanisms (1, 2) in order to serve adjacent tiles simultaneously (or at least faster). This cacheing functionality is lost when running the tileing logic in Lambda, since each request might be served by a different container, which means that each tile read must wait for the file lock to be released before locking the file itself, transforming the simultaneous reads that are made possible by GDAL into a single threaded operation.

One way to get around these issues is to implement intelligent cacheing in the lambda function. (Note: It's important not to directly cache the result of an API call with the associated URL parameters, since some of the parameters only affect the display options of the tile, but rather cache the actual data that gets pulled from S3 - this way a single cached tile can serve multiple requests with different visualization parameters)

If partners accept the (relatively small) cost of ECS services at rest, since ECS has to have at least one EC2 instance running at any given time, and if the scaling rules are well defined, ECS can be favorable for the reasons listed above: better use of multi-threading, re-using database connections, ability to handle larger data tiles/longer runtime operations, etc without timing out.

For all the reasons above, ECS seems to be a better option for the tiling API. If the STAC API requests are small, self contained, and not memory intensive, it may make sense to implement the STAC as a Lambda + API Gateway stack. In this case, it might make sense to have 2 separate FastAPI app's (the tiler deployed as ECS + ELB and the STAC API deloyed at API Gateway + Lambda). Although this has the obvious disadvantage of requiring 2x as much infrastructure to manage.

A distinct advantage of API Gateway is the out-of-the-box integration with cognito user pools for user authentication/authorization (which would ensure that unauthorized users cannot write or edit records to the STAC database). I believe it might also be possible to integrated API Gateway with an ELB.

Ingestion pipeline:

The other large part of the backend is an ingestion pipeline that allows scientists to upload their data. At a minimum, this pipeline should do the the following for any uploaded datafile:

• Validate that the file contains is a properly formatted COG (rio coego validate), and has a nodata value set
• Create a STAC record in the database with the metadata + s3 location of the file

With successive iterations, this pipeline can add additional processing options:

• Convert from GeoTIFF to COG
• Convert from NetCDF to COG
• Unzip archives of COG/GeoTIFF's before converting (if necessary) and ingesting
  (In all three of the above cases, the ingestion mechanism should also copy the original, unprocessed data to an archive for safekeeping/re-ingestion)
• Restrict access for each scientist/science group to a single "delivery" subfolder
• Notify uploading users if their data has been rejected
• Enable "B.Y.O.D" (Bring Your Own Dockerfile) ingestion pipeline where scientists setting up the pipeline can configure any number of different ingestion triggers, where each trigger would execute a customizable processing by packaging the user's Dockerfile as the processing lambda function.

Architecture:

The proposed architecture for the ingestion pipeline would be:

• S3 trigger attached to the /delivery subfolder of the S3 bucket that writes the key of each new file to an SQS Queue
• The SQS Queue invokes a processing lambda

Some advantages of this architecture:

• It's easy to stop the ingestion pipeline from running by removing the SQS event source from the lambda function (1 line to comment) and re-deploying the CDK stack. In case, any delivered files will be added to the queue, and will stay there until the event source is re-attached to the lambda at which point the ingestion will be back-filled.
• SQS will scale up the number of simultaneous lambda invocations, as long as there are no error statuses being returned by the processing lambda, and there is available lambda concurrency, meaning that the data ingestion pipeline won't consume lambda concurrency that is otherwise needed for other applications (since data ingestion doesn't have to be immediate).

Potential improvements to the architecture:

• The SQS records may contain several (probably up to a dozen or so) records of newly created files. In the case of CMIP6 daily data, for example, 12 files represented ~4400 COG's to generate, which was sometimes exceeding the lambda's 15 min timeout. A way to mitigate this would be to implement a "fan-out" lambda, whose job is to invoke the processing lambda over each.
• A second way to mitigate the above issue is to have the "fan-out" logic orchestrated by a stepfunction

A note on repository structure:

Q: Should all three backend services (tiling API, STAC API and ingest pipeline) be their own separate repositories?
A: The fact that this issue is in a repo called delta-backend seems to indicate that we've answered no to this question, but I'm not convinced. The answer to this question is also affected by the decision of how we implement the tiling API and STAC API (ie: are both separate FastAPI apps? Is the TiTiler added as an extension to stac-fastapi? Does titiler-pgstac provide a STAC API endpoint?

As we answer these questions, we should have a clearer idea as to how to structure the backend (1 vs 3 github repos). Some things to keep in mind (or to debate):

• Users should be able to deploy the entire backend with a single CDK deploy command
• If users don't need a STAC API they shouldn't have to fork a repo containing one and have to deploy it
• Since users will have to fork the delta-config repo anyways, in order to be able to set up their own instance of the delta-dashboard, any further customization or configuration should happen through that repo, as much as possible. Ideally we would want to avoid a situation where the user has to fork addition repositories in addition to the delta-config

A note on CDK structure:

Following some of these articles:

• https://docs.aws.amazon.com/cdk/latest/guide/best-practices.html
• https://aws.amazon.com/blogs/developer/recommended-aws-cdk-project-structure-for-python-applications/
  on structuring CDK applications, I'd like to implement the following:

CDK stacks are composable (ie: a single stack can be made up of other stacks). I would like the backend to be deployed as a single stack, but composed of sub-stacks, that are each individually deployed on their as well. Arbitrarily assuming that all three component stay in the same repo for the sake of this example, an example file structure would look like:

tiler_api/
  |_ setup.py  # runtime libs + CDK libs for needed to deploy   
  |_ stack.py # CDK Stack for the TiTiler app
  |_ runtime/
      |_ api/
        |_ main.py # fastapi app definition that imports TiTiler and adds it to the routes    
        |_... # other fastapi related files
stac_api/
  |_ setup.py # runtime libs  + CDK libs for needed to deploy
  |_ stack.py # CDK Stack for the STAC API
  |_ db/
    |_ construct.py # CDK file that generates a Construct for the RDS instance
  |_ bootstrapper/
    |_ infrastructure/ 
      |_construct.py # CDK file that generates a Construct for the Lambda function that bootstraps the postgres DB
      |_ Dockerfile 
    |_ runtime/
      |_ handler.py
ingest_pipeline/
  |_ setup.py # runtime libs + CDK libs for needed to deploy
  |_ stack.py # CDK Stack for the ingest pipeline (SQS Queue + DeadLetter Queue + S3 Bucket + Trigger)
  |_ processing_lambda/
    |_ runtime/
      |_ handler.py
      |_ ... # other supporting files for processing lambda
  |_ infrastructure/
    |_ Dockerfile
    |_ construct.py
app_stack.py

The main aspects are:

• app_stack.py generates a single app composed of each of the constituent stacks, in order to deploy the entire backend with a single CDK deploy
• Each component (tiler API, STAC API, ingest pipeline) has it's own stack.py, which enables the deployment of that component on its own
• If the component (tiler API, STAC API, ingest pipeline) is simple enough, it only needs the stack.py and a runtime/ folder for any runtime code
• If the component is more complex, it is broken up into sub-components. Each sub-components has file called construct.py that exposes a construct for that sub-component. If the sub-component requires runtime code (such as a lambda function), that goes into a folder called runtime/ . If the component's construct requires external resources (such as a Dockerfile) then construct.py file gets saved in the infrastructure/ sub-folder, along with any other files needed for the construct.

cc/ @anayeaye @abarciauskas-bgse @olafveerman @drewbo

Review the referenced csdap-orders files, stack/app_stack.py and stack/constructs/bootstrapped_db.py, along with the stac-fastapi Dockerfile and lambda-handler approach. Resolve the open deployment, database bootstrap, service-boundary, and repository-structure questions, then document a decided architecture and implementation plan.