Merge existing workflows implementation into master

We have three branches with serverless workflows: original workflows, a newer branch workflow-benchmarks-updated that is up-to-date with branch, and feature/workflows

• Finish linting
• Support HTTP trigger for AWS step functions. Docs; issue
• Pull Azure's error from URi
• Fix HTTP's trigger implementation to have a separate path for Azure
• Request ID for Azure - why is it generated?
• What is the purpose of MY_FUNCTION_NAME on Gcp?
• Proper naming of all functions involved in a workflow; keep the same format
• Clean up usage of request id - how is it passed and is it always clean? Split between request id of workflow and functions?
• Avoid reuploading the same code package when sharing function on Google Cloud (possibly also AWS).
• Test container deployment
  • Apply Redis update of connection data for containerized functions
  • Move Redis into envs
• Clean up usage of payload
  • Verify that we don't get double nesting of payload in functions; confirmed in the choice state after failure. There, we get a double-nested payload that we extract.
  • We should safely pass errors without worrying of the nesting.
  • If AWS doesn't all allow restructuring the JSON with an optional error field without involving intrinsic functions or additional states, then we should implement it ourselves.
• Restructurize
  • Workflow should be a module of benchmark (storage, nosql).
  • Rename benchmark directories with better numbers
  • Move workflow classes (generator, workflow) to subdirectories
• Failure handling
  • Implement AWS
  • Implement Azure
  • Implement GCP
• Handling of Redis config
  • CLI script with auth
  • Don't do replacement of vars, just proper envs
  • Ensure that Redis config is provided
  • Move downloading results to proper class
  • Move Redis download to processing
• Any change in the workflow redeploys all of the functions on AWS -> any way to make it more efficient?
• Licensing for code and data.
• Data source documentation
• Add download of data for the 1000-genomes workflow; https://github.com/pegasus-isi/1000genome-workflow and ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/
• Docs
  • Workflow definition for each platform and mapping
  • Workflow sources
  • Workflow data
  • Workflow implementation
  • Redis and profiling

Start by comparing the workflows, workflow-benchmarks-updated, and feature/workflows branches, since no specific files or tests are named. Break the checklist into separately scoped workflow, platform, Redis, deployment, restructuring, and documentation tasks; completion is not defined until those open items are assigned clear outcomes.