apache / apache/ignite

C++ thin Client (2.17.0) - Slow write throughput

Open
#12,212 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

I have my local Apache Ignite setup with 2 nodes that interact with Java and C++ engines.
I am experiencing extremely slow write, where I pick jobs in a loop, process data
Every time, each job writes ~1320000 records in Ignite.
I am using PutAll() in a batch of 3000 records with 2 threads in a C++ based application.
4 such applications write to ignite simultaneously the same number of records.

Specs of machine:
memory=25GB
processors=18
swap=12GB

**Question: Is it possible to improve the performance of my local setup in terms of the time taken to complete a job? Am I using inefficient structures or inefficiently inserting the records? Initially, when it takes about 309 to do the insertion, I cannot understand why the time is exponentially increasing for subsequent jobs.**

The performance measured is
```
- Job 1: Time taken to write results to Ignite: 309seconds.

- Job 2: Time taken to write results to Ignite: 343 seconds
- Job 3: 469 seconds
- Job 4:760 seconds
- Job 5; 945 seconds
```

I am running this whole setup in docker where my ignite config looks like :
```












































ignite-1:47500..47509
ignite-2:47500..47509







```

**Sample code (for reference only, doesn't compile):**

```const size_t BATCH_SIZE = 2500;
const int NUM_THREADS = 2;
const int maxRetries = 3;
const std::chrono::milliseconds retryDelay(1000);
if (analytic_name == "MYANALYTIC") {
std::map batch;

for (size_t i = startIdx; i < endIdx; ++i) {
const auto& v = allData[i];
if (v.size() < 13) continue;
```
std::string uniqueId = groupId + "_" + std::to_string(i);
ResultKey key(uniqueId, v[0]);
ResultValue val(v[1], v[2], std::stod(v[3]), std::stod(v[4]),
v[5], std::stod(v[6]), v[7], std::stod(v[8]),
v[9], std::stod(v[10]), v[11], v[12]);

batch[key] = val;
if (++rowsWritten % BATCH_SIZE == 0) {
for (int attempt = 0; attempt < maxRetries; ++attempt) {
try {
myCache.PutAll(batch);
batch.clear();
break;
} catch (...) {
std::this_thread::sleep_for(retryDelay);
}
}
}
}
if (!batch.empty()) {
for (int attempt = 0; attempt < maxRetries; ++attempt) {
try {
myCache.PutAll(batch);
break;
} catch (...) {
std::this_thread::sleep_for(retryDelay);
}
}
}
}
```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Ignite XML configuration and the sample C++ PutAll loop, then reproduce the increasing write times across successive jobs. Profile persistence, batching, retries, and concurrent writers while checking the two-node Docker setup. Done would require identifying a specific bottleneck and demonstrating an agreed performance improvement, but the issue provides no benchmark target or code files.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker, java
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.