[Java] serialize to output stream is limited to 2GB
- Dominant language
- Java
- Stars
- 4.5k
- Forks
- 443
- Avg merge
- 6h 3m
- Merged PRs (30d)
- 74
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-fury/issues) and found no similar issues.
### Version
Version: 0.4.1
OS: Windows
JDK: 21
### Component(s)
Java
### Minimal reproduce step
```java
public static void main(String[] args) throws Exception {
Fury fury = Fury.builder().requireClassRegistration(false).build();
try (OutputStream output = new BufferedOutputStream(Files.newOutputStream(Files.createTempFile(null, null)))) {
fury.serialize(output, new BigObj());
}
}
static public class BigObj {
public byte[] b1 = new byte[Integer.MAX_VALUE/2];
public byte[] b2 = new byte[Integer.MAX_VALUE/2];
}
```
### What did you expect to see?
I was hoping to get a file with 2147483646 bytes, all zero.
### What did you see instead?
```java
Exception in thread "main" java.lang.NegativeArraySizeException: -2147483510
at io.fury.memory.MemoryBuffer.ensure(MemoryBuffer.java:1980)
at io.fury.memory.MemoryBuffer.writePrimitiveArrayWithSizeEmbedded(MemoryBuffer.java:1946)
at io.fury.serializer.ArraySerializers$ByteArraySerializer.write(ArraySerializers.java:290)
```
### Anything Else?
I think when providing an OutputStream to the serialize method the intermediate MemoryBuffer should behave like the buffer inside the BufferedOutputStream. When the buffer is full it should flush its content to the underlying OutputStream in order to free up its bytes.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Assessment
This issue has not been assessed yet.