Applying ops under load
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 152
- Forks
- 68
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 3
Description
Hi Guys!
About the project: it's a quill collaborative rich text editor. On the back-end side, there is a load balancer over 5 instances of sharedb connected through redis. Sharedb-mongo - is the db adapter.
I'm doing a kind of load testing. I'm running end-to-end tests which simulate users editing - each user produces changes with some frequency. I'm limited in resources so I'm able to run about 50 chromedriver instances (so 50 users). As I have 5 instances of sharedb I expect 5 users per document working concurrently. So these tests simulate 50 users typing within 10 documents. The consumed CPU and RAM look good. When I noticed that under such a load, synchronizing users takes time, I added logs to trace time of the applying op. And I see that applying an op mostly takes milliseconds. BUT sometimes it takes even more than 10 seconds - it's a case when "commit" is unsuccessful and sharedb has to re-fetch snapshot/ops (to re-transform the op). Most of this time is spent on accessing mongo.
Is there anything that I can improve? Or is it a limit?
I use only one mongo collection - will using multiple collections improve the experience?
Applying op timing:
`
+++ getSnapshot before 0
### collection is get 0
### beforeSnapshotLookup middlewate 0
### snapshot is fetched 0.001
+++ getSnapshot after 0.001
~~~ apply 0.001
~~~ commit 0.001
~~~ afterWrite 0.004
+++ getSnapshot before 0
### collection is get 0
### beforeSnapshotLookup middlewate 0
### snapshot is fetched 0.001
+++ getSnapshot after 0.001
~~~ apply 0.001
~~~ commit 0.001
~~~ afterWrite 0.003
+++ getSnapshot before 0
### collection is get 0
### beforeSnapshotLookup middlewate 0
### snapshot is fetched 0.851
+++ getSnapshot after 0.851
+++ getOpsToSnapshot before 0.851
### OP collection is get 0.851
### ops are fetched 1.774
+++ getOpsToSnapshot after 1.774
+++ ops transformed and will be applied 1.774
~~~ apply 1.774
~~~ commit 1.774
+++ unsuccessful commit -> do retry
+++ getSnapshot before 2.15
### collection is get 2.15
### beforeSnapshotLookup middlewate 2.15
### snapshot is fetched 2.152
+++ getSnapshot after 2.152
+++ getOpsToSnapshot before 2.152
### OP collection is get 2.152
### ops are fetched 2.154
+++ getOpsToSnapshot after 2.154
+++ ops transformed and will be applied 2.154
~~~ apply 2.154
~~~ commit 2.154
~~~ afterWrite 2.241`
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the adapter paths represented by the logs: getSnapshot, getOpsToSnapshot, commit, and afterWrite. Reproduce the reported load with 50 users across 10 documents and inspect MongoDB timings when a commit is retried. Done means identifying whether the delay is adapter behavior or an external limit, with evidence from the load test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, mongodb, redis
- Domain
- backend, database, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100