Proposal: Serverless Durable Workflow

## Prerequisites

* [x] I am running the latest version. (`up upgrade`)
* [x] I searched to see if the issue already exists.
* [x] I inspected the verbose debug output with the `-v, --verbose` flag.
* [x] Are you an Up Pro subscriber?

## Description

### First of all - Thank you very much

for creating and maintaining Up!

### This is a feature request

I have never submitted this kind of proposal to a "serverless" framework/platform due to its limited scope. But now, I'm wondering if this fits within the scope of Up - which is relatively more "high-level" than other serverless frameworks, afaics.

### Problem

One of gotachas going serverless is that how we could implement a durable workflow.
A durable workflow is a set of orchestrated, durable jobs to achieve a bigger goal not fit within a single request/response. Typically a fan-in/fan-out pattern is involved in a workflow.

Example use-cases of durable workflows include:

- Generating multiple images in parallel, to be concatenated into a single tarball,
- Generating large PDFs faster by rendering pages in parallel,
- Send e-mails to multiple recipients in parallel,

and so on.

### What's a durable workflow?

You may already know but -

A durable job persists across function/method/process/container/node/cluster failure.

A job must be durable. Otherwise, we have to implement our own durable job runner on top of the given platform, almost certainly utilizing durable datastores/message queues.

### What I think serverless platform/frameworks could help?

In theory, "serverless" fits this use-case nicely, because it provides us massive horizontal scalability.

However, in today's serverless world, it is very cumbersome to actually implement one. Concretely speaking, we all had been implementing this with a workflow engine like Luige, Airflow, AWS Step Functions, in-house job runner with (sort of) durable message queues like AWS SQS/datastore like DynamoDB, "normal" RDBMS like MySQL, Postgres, and so on calling idempotent serverless functions with retries. Some cloud providers do seem to provide their own durable workflow system like [Durable Functions in Azure](https://docs.microsoft.com/en-us/azure/azure-functions/durable-functions-overview).

How helpful it would be if we could implement a durable workflow on top of a serverless platform with less boilerplate, less coupling to underlying datastores/message queues/libraries?

### What I think Up could help?

Up already achieves a clean abstraction on top of AWS Lambda, cleverly adapting a vanilla HTTP server into a single serverless function, which is nice.

It is also a huge step forward for us to stop building distributed monoliths by splitting serverless functions too much. A microservice(I like buzzwords 😄) should be self-contained. Splitting a microservice into pieces which aren't reusable by their own doesn't provide us real benefits.

Alas - I wish I could build an even more self-contained serverless app - self-contained in a sense that it contains not only a full http server app, but also a durable workflows augmenting it.

### Implementation idea

I really don't want to prevent you inspiration(without it, Up isn't here today. Really good job!) with my rough-edged ideas, but let me share purely for sync up.

To define a workflow, we might need (1)a DSL to describe a DAG of tasks per workflow and (2)implementations of each idempotent task.

IMHO, we can use our http server app to just handle (2), similar to what Up is going to do with cron/scheduled job #379 and what GAE had been doing.

Regarding (1), it would be nice if I could write my "up.json" like the below to turn my regular HTTP server app into a durable workflow runner:

```json
{
"name": "app",
"workflow": {
"myjob": {
"trigger": {
"request": {
"path": "/myjob"
},
},
"tasks": {
"entry": {
"request": {
"path": "/calc/copy"
},
"next": [
"add1",
"add2"
],
"description": "fan-out the input to two tasks: add1 and add2. as the task is reffered from nowhere, it is the entry point of the job. you make request to /calc/copy against the API gateway endpoint to start an instance of this job."
},
"add1": {
"request": {
"path": "/calc/add"
},
"next": [
"mul"
],
"description": "somehow sum inputs. the spec of inputs/outputs is completely up to the http handler for /calc/add"
},
"add2": {
"request": {
"path": "/calc/add"
},
"next": [
"mul"
],
"description": "equivalent to add1, but with a different name. the task names must be unique in order to build a DAG of tasks. if you had two tasks named `add`, it isn't a DAG."
},
"mul": {
"request": {
"path": "/calc/mul"
},
"description": "multiply inputs. as the task has no `next` tasks, it is the final task"
}
}
}
}
}
```

Benefits:

- Each task can be tested independently with regular http clients like wget, curl
- Each task can be tested similarly as regular http handlers
- In/outs of each task can be fully customizable without bloating the DAG in up.json. Just parse request bodies and send responses in JSON or alike.
- Authn/Authz can be done for workflow tasks with regular auth proxy/middlewares for http server.
- API gateway to call AWS step function to call Up-managed AWS lambda function seems straightforward to implement.
- Could support multiple backends. Fits the direction of Up to support multiple clouds/platforms.

Downsides:

- Yet another DSL. Forcing us to learn yet another DSL to describe DAGs may complicate things!
- No way to support dynamic DAGs like Airflow. It uses a regular programming language(python) to describe DAG.

// This is already a very long read so I'm quitting here anyway. Thanks for reading!

## Love Up?

Definitely.
I especially like the U/X of Up including dope CLI, concise docs, friendly author 😃

I've already donated & subscribed to Pro for personal-use,
and am really looking forward to use it for my paid work(s), hopefully in production!