elastic / elastic/logstash

`org.logstash.ackedqueue.Queueable` Interface Should be Rethought?

Open
#7,039 5 comments 2 reactions 1 assignee Claimed by @andrewvc View on GitHub
discuss performance improvements
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

The current `Queuable` interface looks like it should be refactored:

We currently have:

```java

byte[] serialize() throws IOException;

static Object deserialize(byte[] bytes)
```

The serialize method is problematic because it forces exposing internal state or allocating a new `byte[]` for every write.

Since we seem to be using the resultant `byte[]` exclusively via putting them into `ByteBuffer` it seems like:

```java
void writeTo(ByteBuffer buffer) throws IOException;
```

would be a much faster approach, not requiring any allocation of a `byte[]`.

Same goes for:

```java
static Object deserialize(byte[] bytes)
```

It looks like you'd rather want something along the lines of:

```java
Object readFrom(ByteBuffer buffer)
```

This would allow you to simply deserialize from underlying `ByteBuffer` or MMFiles by setting limit and position accordingly and remove the need for very expensive things like this:
`org.logstash.ackedqueue.io.wip.MemoryPageIOStream#read(int)`

```java
private SequencedList read(int limit) throws IOException {
List elements = new ArrayList<>();
List seqNums = new ArrayList<>();

int upto = available(limit);
for (int i = 0; i < upto; i++) {
long seqNum = readSeqNum();
byte[] data = readData();
skipChecksum();
elements.add(data);
seqNums.add(seqNum);
}
return new SequencedList<>(elements, seqNums);
}
```

Also and especially under high load, this approach seems like it could lead to very problematic behavior from GC issues.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.