apple / apple/foundationdb

Question: configuration, tooling and reproduction script behind the Performance documentation numbers

Open
#13,813 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Hi FoundationDB team,

First, thank you for maintaining such a well-documented project. We've been benchmarking FoundationDB on our own hardware and are trying to line our results up against the numbers published on the official Performance page (https://apple.github.io/foundationdb/performance.html), but we're having trouble reproducing them and would be grateful for some clarification.

The page documents the hardware clearly (e.g. 8.2M ops/sec on a 24-machine EC2 c3.8xlarge cluster, 384 server processes, 90% read / 10% write, 16-byte keys and 8-100 byte values; and the latency figures on a 12-machine E3-1240 / SATA SSD cluster with 48 processes). However, a few details that matter for reproduction don't seem to be covered:

1. **Benchmark tool** - Which tool was used to generate the load? Was it a workload defined inside FDB's simulation/test harness, `mako`, a custom client, or something else?
2. **Client-side configuration** - How many load-generating client machines/processes/threads were used, and what concurrency (in-flight transactions) per client? Ops/sec is very sensitive to this and it isn't stated.
3. **Cluster configuration** - What redundancy mode and storage engine (ssd/ssd-2/redwood) were used, and were any non-default knobs set?
4. **FDB version** - Which release were these numbers measured on?
5. **Reproduction script** - Is there a script or a documented procedure we could run to reproduce these results end-to-end? If one exists in the repo, a pointer would be very helpful.

Our goal isn't to dispute the numbers - we just want to make sure we're measuring the same thing so our own comparisons are fair. Any pointers would be much appreciated. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with the linked Performance page and search the repository for the benchmark tool, configuration, or reproduction procedure mentioned in the issue. Determine whether the documented numbers have a reproducible script and identify the benchmark, client, cluster, version, and storage settings needed to reproduce them; done means the missing details or a procedure are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
databases, documentation, performance
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.