C++ thin Client (2.17.0) - Slow write throughput
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
I have my local Apache Ignite setup with 2 nodes that interact with Java and C++ engines.
I am experiencing extremely slow write, where I pick jobs in a loop, process data
Every time, each job writes ~1320000 records in Ignite.
I am using PutAll() in a batch of 3000 records with 2 threads in a C++ based application.
4 such applications write to ignite simultaneously the same number of records.
Specs of machine:
memory=25GB
processors=18
swap=12GB
**Question: Is it possible to improve the performance of my local setup in terms of the time taken to complete a job? Am I using inefficient structures or inefficiently inserting the records? Initially, when it takes about 309 to do the insertion, I cannot understand why the time is exponentially increasing for subsequent jobs.**
The performance measured is
```
- Job 1: Time taken to write results to Ignite: 309seconds.
- Job 2: Time taken to write results to Ignite: 343 seconds
- Job 3: 469 seconds
- Job 4:760 seconds
- Job 5; 945 seconds
```
I am running this whole setup in docker where my ignite config looks like :
```
ignite-1:47500..47509
ignite-2:47500..47509
```
**Sample code (for reference only, doesn't compile):**
```const size_t BATCH_SIZE = 2500;
const int NUM_THREADS = 2;
const int maxRetries = 3;
const std::chrono::milliseconds retryDelay(1000);
if (analytic_name == "MYANALYTIC") {
std::map batch;
for (size_t i = startIdx; i < endIdx; ++i) {
const auto& v = allData[i];
if (v.size() < 13) continue;
```
std::string uniqueId = groupId + "_" + std::to_string(i);
ResultKey key(uniqueId, v[0]);
ResultValue val(v[1], v[2], std::stod(v[3]), std::stod(v[4]),
v[5], std::stod(v[6]), v[7], std::stod(v[8]),
v[9], std::stod(v[10]), v[11], v[12]);
batch[key] = val;
if (++rowsWritten % BATCH_SIZE == 0) {
for (int attempt = 0; attempt < maxRetries; ++attempt) {
try {
myCache.PutAll(batch);
batch.clear();
break;
} catch (...) {
std::this_thread::sleep_for(retryDelay);
}
}
}
}
if (!batch.empty()) {
for (int attempt = 0; attempt < maxRetries; ++attempt) {
try {
myCache.PutAll(batch);
break;
} catch (...) {
std::this_thread::sleep_for(retryDelay);
}
}
}
}
```
Contributor guide
Research direction
Start with the supplied Ignite XML configuration and the sample C++ PutAll loop, then reproduce the increasing write times across successive jobs. Profile persistence, batching, retries, and concurrent writers while checking the two-node Docker setup. Done would require identifying a specific bottleneck and demonstrating an agreed performance improvement, but the issue provides no benchmark target or code files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, docker, java
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100