Scalability issue for small chunk size
- Dominant language
- C
- Stars
- 1.1k
- Forks
- 346
- PR merge metrics
- No merged PRs in 30d
Description
When the data chunk size is small, and the number of data chunks is small (for example, 3 data chunk, 1 parity chunk, chunk size 128B), the performance is really bad, what's worse is that when using in multithreading, the overall aggregated performance is worse than single thread.
Here are the results using aws c5.2xlarge instance (AVX512 support)
chunk size = 128 B, single thread throughput is 1223 MB/s, while multithreading (8 threads) gives around 720 MB/s (aggregated throughput), multiprocessing in this case gives close to 4000 MB/s.
chunk size = 1024 B, single thread throughput is 6715 MB/s, while multithreading gives around 6000 MB/s (aggregated throughput).
Is it because there are some internal state/lock that prevent multithreading to scale?
Contributor guide
Assessment
This issue has not been assessed yet.