apache / apache/arrow-java

[Java] How do I read encrypted Parquet files using Arrow?

未关闭
#207 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: usage
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

### Describe the usage question you have. Please include as many useful details as possible.

I want to read encrypted Parquet files using Arrow Java API to create an input stream for [DuckDB](https://github.com/duckdb/duckdb).

It works for Parquet files as shown below.

```java
import org.apache.arrow.c.ArrowArrayStream;
import org.apache.arrow.c.Data;
import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
import org.apache.arrow.dataset.jni.NativeMemoryPool;
import org.apache.arrow.dataset.scanner.ScanOptions;
import org.apache.arrow.dataset.scanner.Scanner;
import org.apache.arrow.dataset.source.Dataset;
import org.apache.arrow.dataset.source.DatasetFactory;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.duckdb.DuckDBConnection;
import java.net.URI;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;

URI uri = URI.create("file:/path/to/sample.parquet");

ScanOptions options = new ScanOptions(/*batchSize*/ 32768);
try (BufferAllocator allocator = new RootAllocator();
ArrowArrayStream stream = ArrowArrayStream.allocateNew(allocator);
DatasetFactory datasetFactory = new FileSystemDatasetFactory(allocator,
NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri.toString());
Dataset dataset = datasetFactory.finish();
Scanner scanner = dataset.newScan(options);
ArrowReader reader = scanner.scanBatches();) {

Data.exportArrayStream(allocator, reader, stream);

Class.forName("org.duckdb.DuckDBDriver");
try (DuckDBConnection conn = (DuckDBConnection) DriverManager.getConnection("jdbc:duckdb:")) {
conn.registerArrowStream("testStream", stream);

try (Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT * FROM testStream")) {
// Do stuff with the resultset...
}
}
}
}
```

However, I haven't found how to do it for files encrypted using [Parquet modular encryption](https://github.com/apache/parquet-format/blob/master/Encryption.md). Arrow Python documentation shows some [examples](https://arrow.apache.org/docs/python/parquet.html#parquet-modular-encryption-columnar-encryption), but they don't exist for Java.

### Component(s)

Java

贡献指南

打开贡献指南

调研方向

从所示的 Java 入口点开始—FileSystemDatasetFactory、FileFormat.PARQUET、Scanner 和 DuckDBConnection.registerArrowStream—并将它们与链接的 Parquet 模块化加密文档和 Arrow Python 示例进行比较。完成的标准是确定并记录一种将加密 Parquet 数据读入 DuckDB Arrow 流的 Java 方法,或明确记录该功能不可用。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
databases
Issue 类型
文档
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。