apache / apache/arrow-java

[java] Is there a way to get the length of the data from bytes at a small system cost?

未關閉
#138 2 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Java
星號
94
分支
152
平均合併
3 天 16 小時
30 天內合併 PR
11

描述

### Describe the usage question you have. Please include as many useful details as possible.

[java] Is there a way to get the length of the data from bytes at a small system cost?

Like the arrow doc example, https://arrow.apache.org/docs/java/ipc.html#writing-and-reading-streaming-format

We generate the arrow data to bytes[] and transfer it.

```
ByteArrayOutputStream out = new ByteArrayOutputStream();

// create VectorSchemaRoot
List fields = Arrays.asList(encodedVector.getField());
List vectors = Arrays.asList(encodedVector);
try (VectorSchemaRoot root = new VectorSchemaRoot(fields, vectors)) {

// write data
ArrowStreamWriter writer = new ArrowStreamWriter(root, provider, Channels.newChannel(out));
writer.start();
writer.writeBatch();
writer.end();
}

// gen arrow data as bytep[]
out.toByteArray()
```

I want to get the data length with a small system load(cpu mem).

Now i using this way to get data lengeh:

```
ByteArrayInputStream input = new ByteArrayInputStream(out.toByteArray());
ReadChannel readChannel = new ReadChannel(Channels.newChannel(input));
MessageMetadataResult messageMeta = MessageSerializer.readMessage(readChannel);
RecordBatch recordBatch =
(RecordBatch) messageMeta.getMessage().header(new RecordBatch());
input.close();
readChannel.close();

// get data length
recordBatch.length()
```

Is there documentation to guide me on what to do? thanks~

### Component(s)

Java

貢獻指南

開啟貢獻指南

研究方向

從 issue 中連結的 Java IPC 文件開始,檢查其中展示的 ArrowStreamWriter、VectorSchemaRoot、MessageSerializer、ReadChannel 和 RecordBatch API。將序列化後的 byte[] 長度與 RecordBatch.length() 進行比較,並判斷是否需要現有的低開銷 API 或釐清文件;完成的標準是清楚記錄支援的方法及其開銷。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java
領域
data
Issue 類型
文件
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
需要釐清
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。