apache / apache/arrow-java

[Java] Add API for getBufferSizeFor() with density to BaseVariableWidthVector

未关闭
#389 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: enhancement
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

Following the discussion on https://github.com/apache/arrow/pull/9187.

Proposed API in BaseVariableWidthVector.java:

```java

/**
* Get the potential buffer size for a particular number of records and density.
* @param valueCount desired number of elements in the vector
* @param density average number of bytes per variable width element
* @return estimated size of underlying buffers if the vector holds
* a given number of elements
*/
public int getBufferSizeFor(final int valueCount, double density)
```

The current `getBufferSizeFor(int valueCount)` for BaseVariableWidthVector requires that validity and offset vectors have already been allocated for at least the given `valueCount`. If the aim of this method is to estimate memory usage for a value count, it's not very useful because it can only give sizes for less than or equal value counts in the currently allocated vector.

A better approach for approximating memory usage is to include a density argument, along with value count. Then the buffer estimate does not require the validity and offset vector to have any allocation. This also is inline with `setInitialCapacity(int valueCount, double density)`

NOTE: this API should also be added to BaseLargeVariableWidthVector and possibly BaseRepeatedValueVector(Large) as well.

**Reporter**: [Bryan Cutler](https://issues.apache.org/jira/browse/ARROW-11739) / @BryanCutler

**Note**: *This issue was originally created as [ARROW-11739](https://issues.apache.org/jira/browse/ARROW-11739). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

贡献指南

打开贡献指南

调研方向

从 BaseVariableWidthVector.java 开始,将现有的 getBufferSizeFor(int valueCount) 与 setInitialCapacity(int valueCount, double density) 进行比较。查看所链接 pull request 的讨论,然后检查相同的 API 是否适用于 BaseLargeVariableWidthVector 和 BaseRepeatedValueVector(Large);当所请求的基于密度的 API 在所有适用的向量类中都得到一致定义时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
backend-api-design, data
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。