marklogic / marklogic/marklogic-contentpump
Stream ZipEntry reading of CompressedRDFReader
@vzhang-progress is already working on this.
Since Mar 22, 2019.
- Dominant language
- Java
- Stars
- 37
- Forks
- 32
- Avg merge
- 1h 3m
- Merged PRs (30d)
- 12
Description
CompressedRDFReader now buffers ZipEntrys in java heap to read in zipped file. Instead, it should stream the reading process, which is desired and preferrable. CompressedAggregateReader and CompressedDelimitedTextReader has done the same.
In an attempt to do the above job, I found that there is some class in CompressedRDFReader that closes the ZipInputStream in middle of a job. It is related to a java bug JDK-6539065. For CompressedAggregateReader, because I know it's XMLStreamReader that closed the ZipInputStream in middle of a job, we can bypass the issue by overload ZipInputStream to disable the behavior. However, for CompressedRDFReader, I didn't figure out who closed ZipInputStream.
You can refer to bug:39688 and bug:25918 for more info.
Please investigate. Thanks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.