rethinkdb / rethinkdb/docs

Document best practices with AWS / EBS volumes

Open
#940 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
CSS
Stars
118
Forks
161
PR merge metrics
No merged PRs in 30d

Description

From a conversation on Slack:

When we first benchmarked rethinkdb, we were using standard EBS volumes. At first the throughput was impressive! Then, after a few minutes, the database IOPS dropped off to < 100 writes per second. After investigating, I found the bottleneck in this case was disk IOPS. Later, after reading up on EBS storage a bit, I learned that Amazon allows you to "burst" their standard EBS volumes for some time, but then they will throttle your IOPS significantly. However, they sell these EBS "IOPS provisioned" disks, which let you set a provisioned IOPS throughput, which they can guarantee you'll have. After learning this, we switched up our hardware configuration and have been experiencing expected IOPS performance ever since. If you wanted to discover if your bottleneck has to do with disk IOPS, you can use tools like iostat and iotop to profile things yourself, or, the easy way is to look at this stuff with the Amazon CloudWatch EBS console, in particular, look at the "VolumeQueueLength" metric. This metric indicates how many read/write operations are in the disk's queue for a significant amount of time. Use your judgement and decide whether or not EBS IOPS provisioned disks will fix your particular problem, but for us, it was the right choice.

It would be good to document best practices on dealing with EBS volumes in our documentation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing documentation under the linked PaaS documentation page and the Slack conversation about standard and provisioned EBS volumes. Document the relevant EBS performance considerations, including IOPS monitoring with iostat, iotop, and CloudWatch VolumeQueueLength; done means the best-practice guidance is added to the appropriate PaaS documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.