spcl / spcl/serverless-benchmarks

Fetching external benchmarks data

Open
#320 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
199
Forks
105
PR merge metrics
No merged PRs in 30d

Description

Currently, all data is assumed to be cloned from the benchmarks-data repository or downloaded from a GH release. However, we need to provide additional data that is (a) under a problematic license or (b) too large to fetch from GitHub. In particular, this problem appears in SeBS-Flow.

We should have a fetch system that:

  • Runs after initializing benchmarks-data
  • Runs all custom "fetch" scripts in the benchmarks-data.
  • Clearly communicates this to the user - how much we fetch and from where (we should put size expectations to not surprise users).
  • Adds a hidden marker to skip refetching.

Good place to install this feature: https://github.com/spcl/serverless-benchmarks/blob/master/sebs/utils.py#L737

CC: @lolamaspoli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sebs/utils.py around line 737 and inspect how benchmarks-data is initialized, then review the custom fetch scripts in the benchmarks-data repository. Define the fetch step, user-facing size and source information, and the hidden marker behavior. Done means initialization runs the scripts once and skips them on subsequent runs when the marker is present.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.