apache / apache/parquet-java

InternalParquetRecordWriter flushRowGroupToStore not actually writing to file

未关闭
#3,171 5 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

### Describe the bug, including details regarding any error messages, version, and platform.

Description:
The flushRowGroupToStore method in InternalParquetRecordWriter purports to "Flush mem columnStore to file," but no data is actually written to the file system until the close method is invoked. This behavior is unexpected, as one would anticipate data being flushed.

Expected Result:
Data should be written to the file when flushRowGroupToStore is invoked, effectively flushing the in-memory column store contents to the file.

Actual Result:
File contents remain empty until the close method is called, where parquetFileWriter.end is invoked to write the data.

Version : parquet-hadoop 1.13.1

### Component(s)

_No response_

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先阅读 InternalParquetRecordWriter,尤其是 flushRowGroupToStore 和 close,然后跟踪 parquetFileWriter.end 如何写入缓冲的数据。确认预期的 flush 边界,并添加或更新对 flushRowGroupToStore 之后文件内容的覆盖;当 close 被调用之前文件已包含被 flush 的 row group 时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。