intel / intel/isa-l

Scalability issue for small chunk size

Open
#58 5 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
C
Stars
1.1k
Forks
346
PR merge metrics
No merged PRs in 30d

Description

When the data chunk size is small, and the number of data chunks is small (for example, 3 data chunk, 1 parity chunk, chunk size 128B), the performance is really bad, what's worse is that when using in multithreading, the overall aggregated performance is worse than single thread.

Here are the results using aws c5.2xlarge instance (AVX512 support)

chunk size = 128 B, single thread throughput is 1223 MB/s, while multithreading (8 threads) gives around 720 MB/s (aggregated throughput), multiprocessing in this case gives close to 4000 MB/s.

chunk size = 1024 B, single thread throughput is 6715 MB/s, while multithreading gives around 6000 MB/s (aggregated throughput).

Is it because there are some internal state/lock that prevent multithreading to scale?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.