apache / apache/fory

[Java] serialize to output stream is limited to 2GB

Open
#1,528 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.5k
Forks
443
Avg merge
6h 3m
Merged PRs (30d)
74

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-fury/issues) and found no similar issues.

### Version

Version: 0.4.1
OS: Windows
JDK: 21

### Component(s)

Java

### Minimal reproduce step

```java
public static void main(String[] args) throws Exception {
Fury fury = Fury.builder().requireClassRegistration(false).build();
try (OutputStream output = new BufferedOutputStream(Files.newOutputStream(Files.createTempFile(null, null)))) {
fury.serialize(output, new BigObj());
}
}

static public class BigObj {
public byte[] b1 = new byte[Integer.MAX_VALUE/2];
public byte[] b2 = new byte[Integer.MAX_VALUE/2];
}
```

### What did you expect to see?

I was hoping to get a file with 2147483646 bytes, all zero.

### What did you see instead?

```java
Exception in thread "main" java.lang.NegativeArraySizeException: -2147483510
at io.fury.memory.MemoryBuffer.ensure(MemoryBuffer.java:1980)
at io.fury.memory.MemoryBuffer.writePrimitiveArrayWithSizeEmbedded(MemoryBuffer.java:1946)
at io.fury.serializer.ArraySerializers$ByteArraySerializer.write(ArraySerializers.java:290)
```

### Anything Else?

I think when providing an OutputStream to the serialize method the intermediate MemoryBuffer should behave like the buffer inside the BufferedOutputStream. When the buffer is full it should flush its content to the underlying OutputStream in order to free up its bytes.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.