apache / apache/arrow-java

[Java][C] sliced RecordBatch offset info is lost when imported from c-data

未关闭
#88 6 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

### Describe the bug, including details regarding any error messages, version, and platform.

Reproduced on latest arrow release (16.0)

When importing a sliced RecordBatch from c to java

On c side:
```
auto sliced_record_batch = original_record_batch->Slice(/*offset=*/8, /*length=*/2);
arrow::ExportRecordBatch(sliced_record_batch, arrow_array_ptr);
```

On java side:
```
ArrowArray arrowArray = ArrowArray.allocateNew(allocator);
Data.importIntoVectorSchemaRoot(allocator, arrowArray, vectorSchemaRoot, null);
```

The imported vectorSchemaRoot maintains the correct length(which is 2), but the offset info (which is 8) is not respected, hence the content of the imported vectorSchemaRoot points to the first 2 rows of the original_record_batch, while the desired content is sliced_record_batch.

I'm not familiar with arrow code, but it seems that the offset info is actually present in org.apache.arrow.c.ArrowArray.Snapshot, but org.apache.arrow.c.ArrayImporter ignores the offset in org.apache.arrow.c.ArrayImporter.doImport(ArrowArray.Snapshot)

### Component(s)

Java

贡献指南

打开贡献指南

调研方向

首先阅读 org.apache.arrow.c.ArrayImporter.doImport(ArrowArray.Snapshot) 以及 org.apache.arrow.c.ArrowArray.Snapshot 中的 offset 信息。使用 C 端切片后的 RecordBatch 重现 Java 导入,然后验证导入的 vectorSchemaRoot 包含从 slice offset 开始的行,而不是原始 batch 的前几行。

由索引模型根据 Issue 内容生成。

评估

技术栈
c, java
领域
data-engineering
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。