apache / apache/parquet-java

ParquetWriter.getDataSize NullPointerException after closed

未关闭
#2,037 6 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Java Component: Parquet Priority: Major Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

When I run `ParquetWriter.getDataSize()`, it works normally. But after I call `ParquetWriter.close()`, subsequent calls to ParquetWriter.getDataSize result in a NullPointerException.

```
java.lang.NullPointerException
at org.apache.parquet.hadoop.InternalParquetRecordWriter.getDataSize(InternalParquetRecordWriter.java:132)
at org.apache.parquet.hadoop.ParquetWriter.getDataSize(ParquetWriter.java:314)
at FileBufferState.getFileSizeInBytes(FileBufferState.scala:83)
```

The reason for the NPE appears to be in `InternalParquetRecordWriter.getDataSize`, where it assumes that `columnStore` is not null.

But the `close()` method calls `flushRowGroupToStore()` which sets `columnStore = null`.

I'm guessing that once the file is closed, we can just return `lastRowGroupEndPos` since there should be no more buffered data, but I don't fully understand how this class works.

**Environment**: Linux prim 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Linux

openjdk version "1.8.0_112"
OpenJDK Runtime Environment (build 1.8.0_112-b15)
OpenJDK 64-Bit Server VM (build 25.112-b15, mixed mode)

**Reporter**: [Mike Mintz](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mikemintz)
#### Related issues:
- [Add accessor to ParquetWriter to get current data size](https://github.com/apache/parquet-java/issues/1833) (relates to)

**Note**: *This issue was originally created as [PARQUET-860](https://issues.apache.org/jira/browse/PARQUET-860). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

Start with InternalParquetRecordWriter.getDataSize() and the close path through ParquetWriter.close(), focusing on how columnStore and lastRowGroupEndPos change when the writer is closed. Reproduce the post-close call described in the issue and inspect existing writer tests for suitable regression coverage. Done means getDataSize() no longer throws a NullPointerException after close and returns the intended size.

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。