apache / apache/parquet-java

ParquetWriter.getDataSize NullPointerException after closed

オープン
#2,037 コメント 6 件 リアクション 0 件 担当者 0 名 GitHub で見る
Component: Java Component: Parquet Priority: Major Type: bug
主要言語
Java
スター
3.1k
フォーク
1.6k
平均マージ
3日 12時間
マージ済み PR(30日)
33

説明

When I run `ParquetWriter.getDataSize()`, it works normally. But after I call `ParquetWriter.close()`, subsequent calls to ParquetWriter.getDataSize result in a NullPointerException.

```
java.lang.NullPointerException
at org.apache.parquet.hadoop.InternalParquetRecordWriter.getDataSize(InternalParquetRecordWriter.java:132)
at org.apache.parquet.hadoop.ParquetWriter.getDataSize(ParquetWriter.java:314)
at FileBufferState.getFileSizeInBytes(FileBufferState.scala:83)
```

The reason for the NPE appears to be in `InternalParquetRecordWriter.getDataSize`, where it assumes that `columnStore` is not null.

But the `close()` method calls `flushRowGroupToStore()` which sets `columnStore = null`.

I'm guessing that once the file is closed, we can just return `lastRowGroupEndPos` since there should be no more buffered data, but I don't fully understand how this class works.

**Environment**: Linux prim 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Linux

openjdk version "1.8.0_112"
OpenJDK Runtime Environment (build 1.8.0_112-b15)
OpenJDK 64-Bit Server VM (build 25.112-b15, mixed mode)

**Reporter**: [Mike Mintz](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mikemintz)
#### Related issues:
- [Add accessor to ParquetWriter to get current data size](https://github.com/apache/parquet-java/issues/1833) (relates to)

**Note**: *This issue was originally created as [PARQUET-860](https://issues.apache.org/jira/browse/PARQUET-860). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Start with InternalParquetRecordWriter.getDataSize() and the close path through ParquetWriter.close(), focusing on how columnStore and lastRowGroupEndPos change when the writer is closed. Reproduce the post-close call described in the issue and inspect existing writer tests for suitable regression coverage. Done means getDataSize() no longer throws a NullPointerException after close and returns the intended size.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
data-engineering
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。