Data unavailable in second server after successful insert with Ignite 3
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
We have an issue using Ignite 3 from multiple services. Often data is not available although it should be.
# TL;DR
- A fresh cluster with three nodes running
- A table with a combined primary key and some data column
- Two applications: One (the sender) is writing to Ignite, the second one (the receiver) is reading from Ignite.
- After the sender stored multiple rows with the same `id` but with different sequence numbers, it sends a REST request with the id to the receiver.
- The receiver then tries to select data by the id and doesn't get any records back.
# Setup
This is the SQL we've run in Ignite:
```sql
CREATE ZONE my_zone (PARTITIONS 25, REPLICAS 1, AUTO SCALE UP 10, AUTO SCALE DOWN 600) STORAGE PROFILES ['memStorageProfile'];
CREATE TABLE IF NOT EXISTS data_table(
id VARCHAR(50) NOT NULL,
sequence INT NOT NULL,
data VARCHAR(99999) NOT NULL,
PRIMARY KEY (id, sequence)
) ZONE my_zone STORAGE PROFILE 'memStorageProfile';
```
The corresponding entity class is defined like this:
```java
@Table("data_table")
public class DataTable {
@Id
@Column(value = "id", length = 50, nullable = false)
private String id;
@Id
@Column(value = "sequence", nullable = false)
private int sequence;
@Column(value = "data", nullable = false)
private String data;
// Constructors, getters and setters omitted.
}
```
The sender uses the following code to store data into Ignite:
```java
try (IgniteClient igniteClient = IgniteClient
.builder()
.addresses("127.0.0.1:10800", "127.0.0.1:10801", "127.0.0.1:10802")
.build()) {
Table dataTable = igniteClient.tables().table("data_table");
RecordView recordView = dataTable.recordView(DataTable.class);
String id = UUID.randomUUID().toString();
List data = new ArrayList<>();
for (int i = 0; i < 5; i++) {
data.add(new DataTable(id, i, "data-" + i));
}
recordView.upsertAll(null, data);
// Send rest call to receiver
}
```
The receiver is using the following code to get the data:
```java
IgniteClient igniteClient = IgniteClient
.builder()
.addresses("127.0.0.1:10800", "127.0.0.1:10801", "127.0.0.1:10802")
.build();
Table dataTable = igniteClient.tables().table("data_table");
RecordView recordView = dataTable.recordView(DataTable.class);
// Wait to receive the REST call with the id...
Cursor cursor = recordView.query(null, columnValue("id", equalTo(id)))) {
while (cursor.hasNext()) {
// Do something
}
```
# Actual results
When receiving the REST call (which is sent *after* the `upsertAll` call in our sender returned) we don't receive any data when running `query`, ie the cursor is empty.
I added some retry logic to figure out if data will be available after some time. Here is an example output of that:
```
Data not yet available, waiting 100 ms...
Data not yet available, waiting 200 ms...
Data retrieved successfully after 2 cycles and 300 ms
```
So, it took between 100 and 300ms in this example after the REST request was received on the receiver side until the data was actually readable. Note it sometimes takes longer.
# Expected results
After the `upsertAll` returned, other services can read the data.
# Things we tried
We tried different things to get around this, with no success:
- Wrap the `upsertAll` in a transaction.
- Use a read-only transaction when reading the data.
- Immediately after writing the data, we read them in the sender using the exact same code as the receiver and successfully get the newly inserted data back.
- Instead of a cluster with three nodes, we created a cluster with a single node.
- Replace aimem with aipersist.
Why is that happening and how can we get around this? Any hint is much appreciated. I can also provide the minimal example in case that is helpful. Thanks in advance!
Contributor guide
Assessment
This issue has not been assessed yet.