Old Parquet files with wrong Compressed Size not Readable
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
In certain circumstances, the CLI will fail to read old (perhaps ancient) parquet files that have an incorrect compressed_size field set in the column metadata that does not include the dictionary page (at least according to the comment in the code). The code that is supposed to handle this does not flip the byte buffer it reads the extra bytes into. It appears to have been broken for a few years now.
I have written a PR that includes a defective parquet file with this issue, wrote a unit test that fails without the additional `flip`, and validated that the code works afterwards.
This is a trivial minor issue that was from learning the code rather than actually addressing a production issue, so there's no urgency.
贡献指南
这个仓库没有索引到贡献指南
调研方向
从处理 compressed_size 不包含字典页的旧 Parquet 文件的 CLI 代码路径开始,然后检查用于额外字节的字节缓冲区。使用 issue 中描述的有缺陷的 Parquet 文件和单元测试;当测试通过且缓冲区翻转后旧文件可读时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 缺陷
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 55/100