apache / apache/fory

[Java] enable zero-copy in `serialize(OutputStream)`

Open
#994 1 comment 0 reactions 0 assignees View on GitHub
enhancement java
Dominant language
Java
Stars
4.5k
Forks
443
Avg merge
6h 3m
Merged PRs (30d)
74

Description

## Is your feature request related to a problem? Please describe.
For large graph, current `serialize(OutputStream)` will serialize data into a binary heap buffer, then write it into `OutputStream`, which is not really streaming.

Large graph will have many binary buffers inside, serializing without streaming will introduce an extra copy, and can't overlap network. If we enable fury zero-copy support, all binary buffer will be handled separately without writting it into the heap buffer, which will reduce memory copy, and serialization time. And then we can send in-band buffers and all out-of-band buffers chunk by chunk to speed up the transfer.

## Describe the solution you'd like
Make `serialize(OutputStream)` use bufferCallback to capture binary buffer:
```java
List buffers = new ArrayList<>();
serialize(outputStream, obj, o -> {
buffers.add(o);
return false;
});
buffers.forEach(bufferObject -> {
int size = bufferObject.totalBytes();
byte[] sizeBytes = toBytes(size);
outputStream.write(sizeBytes);
bufferObject.writeTo(outputStream);
});
```

Potential disadvantages:
If all binary buffers in object graph are small objects, then the serialization will be slightly slower.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.