Parquet writer classes don't close underlying output stream in case of errors.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
org.apache.parquet.io.OutputFile interface does not implement Closeable.
In my opinion it implies that created streams are fully managed by parquet-mr classes.
Unfortunately opened stream will not be closed in case of IO or other failure.
There are two places I can find for this problem:
- During writer creation (org.apache.parquet.hadoop.ParquetWriter.Builder#build()) - created stream should be closed if writer creation fails.
- During writer close(org.apache.parquet.hadoop.ParquetWriter#close) - underlying stream should be closed regardless of any faced failures.
Although I didn't examine ParquetReaded that much.
**Reporter**: [Andrei Lopukhov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=alopukhov)
#### Original Issue Attachments:
- [TestExample.java](https://issues.apache.org/jira/secure/attachment/13053139/TestExample.java)
**Note**: *This issue was originally created as [PARQUET-2216](https://issues.apache.org/jira/browse/PARQUET-2216). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.parquet.hadoop.ParquetWriter.Builder#build() and org.apache.parquet.hadoop.ParquetWriter#close(), then review the attached TestExample.java for the failure cases. Trace how the org.apache.parquet.io.OutputFile stream is created and determine whether it is closed when writer creation or close encounters an error. Done means the underlying stream is closed in both reported failure paths, with tests covering those cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100