bchavez / bchavez/RethinkDb.Driver
Investigate ReGrid Performance vs Node
- Dominant language
- C#
- Stars
- 381
- Forks
- 129
- PR merge metrics
- No merged PRs in 30d
Description
Seems like Node ReGrid can get 3x more writes than .NET; yielding faster upload wall time. See image below (credits @buskila):

## Test setup
Upload only:
File Size: 1 GB.
Server: RethinkDB / Linux / Ubuntu 14, 3 nodes
Client: .NET Core / Linux
Chunk Size: Default
Batch Size: Default 8 -> 32
They tried single connection and connection pooling. No difference.
Using Stream IO:
```
// Upload a file using an IO stream
Guid uploadId;
using( var fileStream = File.Open("C:\\video.mp4", FileMode.Open) )
using( var uploadStream = bucket.OpenUploadStream("/video.mp4") )
{
uploadId = uploadStream.FileInfo.Id;
fileStream.CopyTo(uploadStream);
}
```
## Suspicion
Too much chunk calculation in stream upload code. Try to parallelize / simplify some of this, especially when given `byte[]`.
Node's ReGrid upload code is here:
https://github.com/internalfx/regrid/blob/master/lib/upload.js
## Other notes
This should come after #77 is done.
After some discussion with @interalfx (thanks a bunch), the upload code is using node streams. Node streams info via @buskila:
```
Using .pipe() has other benefits too, like handling backpressure automatically so that
node won't buffer chunks into memory needlessly when the remote client
is on a really slow or high-latency connection.
```
https://github.com/substack/stream-handbook
Currently, @internalfx runs 10 network requests in flight at any given time. In a scenario where there is infinite network latency, node won't write to the ReGrid API until at least 1 network request completes.
Cool. I think we could maybe do the same with 10 async tasks laying down bytes over a connection pool then as they complete, then come back read more bytes as network requests complete.
---
## Other Research Findings
## RethinkDB Limitations
- `Query size (419554663) greater than maximum (134217727).`
So batch size can't be too big, Max query size is ~130MB something. So only ~130MB per batch max.
Contributor guide
Research direction
After #77 is complete, start with the Stream IO upload path and compare it with the Node implementation at lib/upload.js. Reproduce the 1 GB upload against RethinkDB on three Ubuntu nodes using the stated batch sizes, accounting for the roughly 130 MB maximum query size. Done means identifying the throughput bottleneck and documenting or addressing the performance difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100