apache / apache/parquet-java

ParquetWriter::close sometimes fail

未关闭
#1,701 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Java Component: Parquet Priority: Major Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

we sometimes run into an exception when closing a ParquetWriter instance:

 
```java

2024-06-10 10:44:01.398    org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
2024-06-10 10:44:01.398        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:85)
2024-06-10 10:44:01.398        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:94)
2024-06-10 10:44:01.398        at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:144)
2024-06-10 10:44:01.398        at org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:437)
2024-06-10 10:44:01.398    Caused by: java.nio.channels.ClosedChannelException: null
2024-06-10 10:44:01.398        at org.apache.hadoop.hdfs.ExceptionLastSeen.throwException4Close(ExceptionLastSeen.java:73)
2024-06-10 10:44:01.398        at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:158)
2024-06-10 10:44:01.398        at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:639)
2024-06-10 10:44:01.398        at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:594)
2024-06-10 10:44:01.398        at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:136)
2024-06-10 10:44:01.398        at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:65)
2024-06-10 10:44:01.398        at org.apache.parquet.hadoop.ParquetFileWriter.close(ParquetFileWriter.java:1663)
2024-06-10 10:44:01.398        at org.apache.parquet.util.AutoCloseables.close(AutoCloseables.java:49)
2024-06-10 10:44:01.398        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:83)
```

**Reporter**: [Roelof Naude](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=naude-r)

**Note**: *This issue was originally created as [PARQUET-2496](https://issues.apache.org/jira/browse/PARQUET-2496). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

以提供的 ClosedChannelException 堆栈跟踪为起点,跟踪从 ParquetWriter.close 经由 InternalParquetRecordWriter.close、ParquetFileWriter.close 和 HadoopPositionOutputStream.close 的关闭路径。确定导致 HDFS 输出通道在 hflush 之前保持关闭状态的条件,然后为间歇性故障添加覆盖,并验证 ParquetWriter 的关闭能够在不抛出此异常的情况下完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
hadoop, java
领域
data-engineering
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
32/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。