jankotek / jankotek/mapdb

Serialization/Deserialization fails for large objects with ArrayIndexOutOfBoundsException

Open
#930 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

3.0
Dominant language
Java
Stars
5.1k
Forks
877
PR merge metrics
No merged PRs in 30d

Description

I have a custom serializer:

class CustomSerializerDesc implements Serializer<DescribeSObjectResult>
	{
		private final XStream xstream = new XStream();

		@Override
		public void serialize(final DataOutput2 out, final DescribeSObjectResult sObject) throws IOException
		{
			xstream.toXML(sObject, out);
		}

		@Override
		public DescribeSObjectResult deserialize(final DataInput2 input, final int available) throws IOException
		{
			final DataInputToStream input2 = new DataInput2.DataInputToStream(input);
			return (DescribeSObjectResult) xstream.fromXML(input2);
		}

	}

where the usage drops across

final DB db = DBMaker
				.fileDB(SalesforceOptions.cachePath.resolve("SalesforceObjects").toFile())
				.fileMmapEnable()
				.fileDeleteAfterClose()
				.cleanerHackEnable()
				.make();
		final CustomSerializerDesc customSerializer = new CustomSerializerDesc();
		final List<DescribeSObjectResult> finalDescs = db.indexTreeList("test", customSerializer).createOrOpen();

However, when iterating through the lists, it throws an exception
Exception in thread "main" with the root cause pointing here:

Caused by: java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at org.mapdb.DataInput2$ByteArray.readFully(DataInput2.java:28)
	at org.mapdb.DataInput2$DataInputToStream.read(DataInput2.java:242)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)

This doesn't happen to small objects. I tried a custom serializer with a different class that works fine. However, this is a large class and somehow fails. Does serialization/deserialization have any limits on the buffer size?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with DataInput2.java, especially ByteArray.readFully and DataInputToStream.read, then reproduce the failure using the custom serializer, file-backed DB, and large DescribeSObjectResult shown here. Determine whether large-object deserialization exceeds a buffer or exposes another read-boundary problem; done means iteration completes without ArrayIndexOutOfBoundsException for the large object.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.