Benchmark the assembly of thrift objects, and possibly create a more efficient ReplayingTProtocol
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
The current implementation of parquet thrift creates an instance of TProtocol for each value of each record and builds a stack of these events, which are then replayed back to the TBase.
I'd be curious to benchmark this, and if it's slow, try building a "ReplayingTProtocol" that instead of having a stack of TProtocol instances, contains a primitive array of each type. As events are fed into this replaying TProtocol, it would just add these primitives to its buffers, and then the TBase would drain them. This would effectively let us stream the values into the TBase without making an object allocation for each value.
The buffers could be set to a certain size, and if they fill up (which they sholdn't in most cases), the TBase could begin draining the protocol until it is empty again, at which point the TProtocol can block the TBase from draining further while the parque record assembly feeds it more events.
This is all moot if it turns out not to be bottleneck though :)
**Reporter**: [Alex Levenson](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=alexlevenson) / @isnotinvain
**Note**: *This issue was originally created as [PARQUET-33](https://issues.apache.org/jira/browse/PARQUET-33). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
首先定位当前的 parquet thrift 记录组装路径,该路径会为每个值创建一个 TProtocol,并在设计更改之前对其进行基准测试。将分配和吞吐量与现有的 event-stack replay 进行比较;当基准测试能够明确决定是否值得引入 ReplayingTProtocol,并且范围和测试已记录时,即视为完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- performance
- Issue 类型
- 重构
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 18/100