apache / apache/ignite

Data unavailable in second server after successful insert with Ignite 3

Open
#12,639 10 comments 0 reactions 1 assignee Claimed by @ptupitsyn View on GitHub
ignite-3
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

We have an issue using Ignite 3 from multiple services. Often data is not available although it should be.

# TL;DR
- A fresh cluster with three nodes running
- A table with a combined primary key and some data column
- Two applications: One (the sender) is writing to Ignite, the second one (the receiver) is reading from Ignite.
- After the sender stored multiple rows with the same `id` but with different sequence numbers, it sends a REST request with the id to the receiver.
- The receiver then tries to select data by the id and doesn't get any records back.

# Setup
This is the SQL we've run in Ignite:
```sql
CREATE ZONE my_zone (PARTITIONS 25, REPLICAS 1, AUTO SCALE UP 10, AUTO SCALE DOWN 600) STORAGE PROFILES ['memStorageProfile'];

CREATE TABLE IF NOT EXISTS data_table(
id VARCHAR(50) NOT NULL,
sequence INT NOT NULL,
data VARCHAR(99999) NOT NULL,
PRIMARY KEY (id, sequence)
) ZONE my_zone STORAGE PROFILE 'memStorageProfile';
```

The corresponding entity class is defined like this:
```java
@Table("data_table")
public class DataTable {
@Id
@Column(value = "id", length = 50, nullable = false)
private String id;

@Id
@Column(value = "sequence", nullable = false)
private int sequence;

@Column(value = "data", nullable = false)
private String data;

// Constructors, getters and setters omitted.
}
```

The sender uses the following code to store data into Ignite:
```java
try (IgniteClient igniteClient = IgniteClient
.builder()
.addresses("127.0.0.1:10800", "127.0.0.1:10801", "127.0.0.1:10802")
.build()) {
Table dataTable = igniteClient.tables().table("data_table");
RecordView recordView = dataTable.recordView(DataTable.class);
String id = UUID.randomUUID().toString();
List data = new ArrayList<>();
for (int i = 0; i < 5; i++) {
data.add(new DataTable(id, i, "data-" + i));
}
recordView.upsertAll(null, data);

// Send rest call to receiver
}
```

The receiver is using the following code to get the data:
```java
IgniteClient igniteClient = IgniteClient
.builder()
.addresses("127.0.0.1:10800", "127.0.0.1:10801", "127.0.0.1:10802")
.build();
Table dataTable = igniteClient.tables().table("data_table");
RecordView recordView = dataTable.recordView(DataTable.class);

// Wait to receive the REST call with the id...

Cursor cursor = recordView.query(null, columnValue("id", equalTo(id)))) {
while (cursor.hasNext()) {
// Do something
}
```

# Actual results
When receiving the REST call (which is sent *after* the `upsertAll` call in our sender returned) we don't receive any data when running `query`, ie the cursor is empty.

I added some retry logic to figure out if data will be available after some time. Here is an example output of that:
```
Data not yet available, waiting 100 ms...
Data not yet available, waiting 200 ms...
Data retrieved successfully after 2 cycles and 300 ms
```
So, it took between 100 and 300ms in this example after the REST request was received on the receiver side until the data was actually readable. Note it sometimes takes longer.

# Expected results
After the `upsertAll` returned, other services can read the data.

# Things we tried
We tried different things to get around this, with no success:
- Wrap the `upsertAll` in a transaction.
- Use a read-only transaction when reading the data.
- Immediately after writing the data, we read them in the sender using the exact same code as the receiver and successfully get the newly inserted data back.
- Instead of a cluster with three nodes, we created a cluster with a single node.
- Replace aimem with aipersist.

Why is that happening and how can we get around this? Any hint is much appreciated. I can also provide the minimal example in case that is helpful. Thanks in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.