New benchmarks and applications

In SeBS, we provide a representative set of functions and have developed a set of serverless workflows that will be included in the upcoming release. However, the serverless field is constantly changing, and new types of applications are being "FaaS-ified". SeBS will benefit from new type of functions, new workflows, and from new applications - the last type has not been considered for SeBS yet.

Functions

The current list of functions is available in the documentation.

New ideas (all should be rather simple to implement thanks to a large number of open-source implementations):

• ML - More ML examples (different Resnet versions, BERT, etc.) - we can use Ml-Perf as a reference; we can use VGG, AlexNet, BERT - import code from MIGnificient benchmarks.
• Utilities - PDF generation and conversion seems to be a common - headless Chrome to generate PDF of a webpage (maybe with Browsershot?), or convert PDFs to different formats (Calibre? pdfkit for Node.js?). Example article with browsershot.. Also possible to generate screenshot of a website.
• Utilities - scanning object storage for viruses; another example.
• Webpage/Utilities - QR code generator.
• URL shortener - example 1, example 2.
• Web scraping - example
• Processing Excel/CSV files - excel file, csv files, DuckDB integration - very interesting; another example and one more.
• Visualization and rendering - Blender example and Mermaid example.
• Two collections of serverless snippets - there might some interesting applications there. ServerlessLand and FunctionResources.
• New scientific applications - examples from Lithops, sentiment analysis and mandelbrot.
• Database queries - deploy SQL database?
• LLM inference - one motivation case, C++ version.
• CI workload - simulate continuous integration, i.e., fetch code, apply linting or type checking, run tests.

Workflows

The current list of workflows is in the PR #88 and in the related thesis. In the PR, we have workflows for MapReduce, video analysis, ExCamera, and ML fitting. The thesis also documents the abstract language we use to specify each workflow.

To extend SeBS, we want to cover new application types and rich workflows with new computational patterns.

Potential new ideas:

• ServerlessLand snippets
• Simple webapp application - todos.
• Another webapp - airline flight booking
• Pywren has several linear algebra applications. It is not written as a workflow, but turning a Cholesky or matrix-matrix multiplication into a workflow with pre-defined schedule would be very interesting.
• AFCL has two interesting benchmarks - we should analyze if they can be expressed properly in our system to run on AWS Step Functions or Azure Durable.
• cbl-translate is a very interesting, rich and complex ML inference workflow that utilize several different models. The workflow uses external APIs such as DeepL, but we should be able to not rely on them - either deploy an existing open-source model for this task instead of an external API or remove that feature entirely.
• maskopy - another interesting utility application that runs on AWS Step Functions.
• Compilation benchmark - examples are in the gg paper. We do not need generality, but an example of offloading compilation steps.

Applications

Our benchmark suite contains functions and workflows, but it does not contain full applications not written as workflows. This can be standalone applications offloading certain tasks to serverless, and fully serverless applications.

• Simple web applications - one motivation example, another one, another one.
• Black-Scholes from PARSEC benchmark suite - we already have code for a serverless invocation that should be integrated soon.
• Microservices are an extremely important workload to the cloud, and several authors have attempted to port them to serverless. This usually requires adding database for state, and replacing RPC with queues and triggers. DeathStarBench is a great source of microservices - SocialNetwork might be a great candidate and it's in C++ for which we have provisional support. Another paper describes porting several microservices - the paper does not come with a code, but some of these application should be open source, e.g., Overleaf.
• Another work is focused on benchmarking applications built on top of serverless triggers - one of the examples might also be new to SeBS and interesting. So far, we have not explored asynchronous triggers in SeBS.
• Serverless Google Maps is open source.
• Simple webapp application - todos.
• Another webapp - airline flight booking
• Pywren has several linear algebra applications.
• cbl-translate is a very interesting, rich and complex ML inference workflow that utilize several different models. We can represent this as an application and remove the dependency on an external ML API.
• Hello-retail is an interesting one. The original repo is gone, but forks are available one and [another](https://github.com/SimonEismann/hello-retail]. Papers using it: 1 and 2
• Data processing - example of parallel processing of large CSV files.

Start with docs/benchmarks.md to understand the existing function list, then review PR #88 and the related thesis for workflow conventions. Choose one narrowly scoped candidate from the proposed functions, workflows, or applications and define its integration boundaries; done should mean the selected benchmark is implemented and represented in the SeBS suite.