apache / apache/arrow-adbc

feat(api): decouple reading partitions from AdbcConnection

Open
#1,537 0 comments 1 reaction 0 assignees View on GitHub
Type: enhancement
Dominant language
C#
Stars
627
Forks
217
Avg merge
22h 22m
Merged PRs (30d)
64

Description

### What feature or improvement would you like to see?

Currently to read data from a partition, using Java as an example:
1. a user must use AdbcStatement.executePartitioned() to get a PartitionedResult containing a list of PartitionDescriptors.
2. For each PartitionDescriptor the user must call AdbcConnection.readPartitioned().

If the user intends to distribute the work across separate processes (or nodes in a distributed system), they must have each process go through the work of creating an AdbcDatabase, building up connection options, then connecting to a node to create a full-fledged AdbcConnection. This could be costly -- for example this may requiring re-running the auth workflow when an auth token was generated already or creating a session object instead of re-using the session already established.

An idea would be to have AdbcDatabase let the caller construct an AdbcPartitionedReader from a PartitionDescriptor instead of requiring the full connection process. The driver implementation can bake all the details on how to connect to and use the node holding the partition (including stateful information such as auth access tokens or session identifiers) so it can skip the potentially heavy-weight connection process.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Java flow from AdbcStatement.executePartitioned() through PartitionedResult and PartitionDescriptor to AdbcConnection.readPartitioned(). Review how AdbcDatabase creates connections and how driver implementations retain connection state. Done means defining and implementing a way to construct a partition reader from a PartitionDescriptor without recreating the full connection workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.