firecracker-microvm / firecracker-microvm/nsdi2020-data

QEMU MicroVM: why such bad results and why not in the Firecracker paper?

Open
#2 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
26
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Dear Firecracker paper authors,

While looking at some other paper (https://github.com/rellermeyer/container_benchmarks/issues/1), I noticed the unexplainably bad results of QEMU's MicroVM in your GitHub repo.

Here is the QEMU MicroVM command line: https://github.com/firecracker-microvm/nsdi2020-data/blob/cf1b30f674523ee44363f74e16bcb301ffb10598/scripts/util_start_qemu_uvm.sh

Here is the commit that added QEMU MicroVM data: https://github.com/firecracker-microvm/nsdi2020-data/commit/b35147fbc1e51754757c6aaad04dcd219b1f45df

You also updated all the plots, some of them include QEMU MicroVM perf, especially this one: https://github.com/firecracker-microvm/nsdi2020-data/blob/master/data/m5d.metal/boot-serial-all.pdf

If I'm reading the plot correctly, and if I'm not confusing two shades of "light blue", then QEMU MicroVM exhibits extremely bad boot time (~600ms, about 2.5x worse than normal QEMU).

I also notice that your final paper https://www.usenix.org/system/files/nsdi20-paper-agache.pdf doesn't mention QEMU MicroVM at all (this may be just timing issues, since your GitHub updates are dated with 9. February and the NSDI conference was on 25. February).

So I have two questions:
- Do you have any explanations on QEMU MicroVM boot times? Did you maybe expand on this in some follow-up paper, video, blog, etc?
- How can you explain the fact that QEMU's MicroVM behaves worse than normal QEMU ( with normal machine type)?
- Why didn't you include any mentions of QEMU's MicroVM in the final paper?

Contributor guide

Open the contributing guide

Research direction

Start with scripts/util_start_qemu_uvm.sh, the commit that added QEMU MicroVM data, and data/m5d.metal/boot-serial-all.pdf; compare the command line with the reported boot-time results. Read the linked NSDI paper and benchmark context, then document an evidence-based explanation for the performance difference and the paper's omission in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
documentation, performance
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.