bchavez / bchavez/RethinkDb.Driver

Investigate ReGrid Performance vs Node

Open
#96 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
381
Forks
129
PR merge metrics
No merged PRs in 30d

Description

Seems like Node ReGrid can get 3x more writes than .NET; yielding faster upload wall time. See image below (credits @buskila):

![pasted_image_at_2016_08_24_13_03](https://cloud.githubusercontent.com/assets/478118/17918557/d28f41ae-697a-11e6-82a6-1215e2701654.png)
## Test setup

Upload only:
File Size: 1 GB.
Server: RethinkDB / Linux / Ubuntu 14, 3 nodes
Client: .NET Core / Linux

Chunk Size: Default
Batch Size: Default 8 -> 32

They tried single connection and connection pooling. No difference.

Using Stream IO:

```
// Upload a file using an IO stream
Guid uploadId;
using( var fileStream = File.Open("C:\\video.mp4", FileMode.Open) )
using( var uploadStream = bucket.OpenUploadStream("/video.mp4") )
{
uploadId = uploadStream.FileInfo.Id;
fileStream.CopyTo(uploadStream);
}
```
## Suspicion

Too much chunk calculation in stream upload code. Try to parallelize / simplify some of this, especially when given `byte[]`.

Node's ReGrid upload code is here:
https://github.com/internalfx/regrid/blob/master/lib/upload.js
## Other notes

This should come after #77 is done.

After some discussion with @interalfx (thanks a bunch), the upload code is using node streams. Node streams info via @buskila:

```
Using .pipe() has other benefits too, like handling backpressure automatically so that
node won't buffer chunks into memory needlessly when the remote client
is on a really slow or high-latency connection.
```

https://github.com/substack/stream-handbook

Currently, @internalfx runs 10 network requests in flight at any given time. In a scenario where there is infinite network latency, node won't write to the ReGrid API until at least 1 network request completes.

Cool. I think we could maybe do the same with 10 async tasks laying down bytes over a connection pool then as they complete, then come back read more bytes as network requests complete.

---
## Other Research Findings
## RethinkDB Limitations
- `Query size (419554663) greater than maximum (134217727).`
So batch size can't be too big, Max query size is ~130MB something. So only ~130MB per batch max.

Contributor guide

Open the contributing guide

Research direction

After #77 is complete, start with the Stream IO upload path and compare it with the Node implementation at lib/upload.js. Reproduce the 1 GB upload against RethinkDB on three Ubuntu nodes using the stated batch sizes, accounting for the roughly 130 MB maximum query size. Done means identifying the throughput bottleneck and documenting or addressing the performance difference.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.