Azure / Azure/Moodle

Gluster FS possibly causing high latency on high workload

Open
#127 7 comments 0 reactions 0 assignees View on GitHub
bug Priority 0 (will address)
Dominant language
Shell
Stars
184
Forks
186
PR merge metrics
No merged PRs in 30d

Description

We've been experiencing high latency with Gluster FS on the [time-gated exam scenario](https://github.com/Azure/Moodle/blob/hs-loadtest-time-gated-exam/loadtest/time-gated-exam-test.jmx). We don't know exactly if Gluster is the issue and why in that case, but replacing Gluster with NFS makes the high latency issue go away, so naturally suspecting Gluster.

We'll need to dive deeper on how Gluster works and why it might cause perf bottleneck like we've been experiencing. In the meantime, we might need to provide an alternative, like HA NFS that's described in places like the following:

- https://www.howtoforge.com/high_availability_nfs_drbd_heartbeat
- https://help.ubuntu.com/community/HighlyAvailableNFS

These are pretty old, and still the top results I get from my related web search, making me think that this is not a widely used solution, but we should evaluate the option. Any PR of an ARM template deploying an HA NFS 2-VM cluster would be highly appreciated.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with loadtest/time-gated-exam-test.jmx and compare the high-latency behavior when using Gluster FS versus NFS. Investigate whether Gluster is the bottleneck, then evaluate whether an ARM template for an HA NFS 2-VM cluster is an appropriate alternative; the work is done when the cause or alternative is documented with supporting results.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, infrastructure, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.