feat(api): decouple reading partitions from AdbcConnection
- Dominant language
- C#
- Stars
- 627
- Forks
- 217
- Avg merge
- 22h 22m
- Merged PRs (30d)
- 64
Description
### What feature or improvement would you like to see?
Currently to read data from a partition, using Java as an example:
1. a user must use AdbcStatement.executePartitioned() to get a PartitionedResult containing a list of PartitionDescriptors.
2. For each PartitionDescriptor the user must call AdbcConnection.readPartitioned().
If the user intends to distribute the work across separate processes (or nodes in a distributed system), they must have each process go through the work of creating an AdbcDatabase, building up connection options, then connecting to a node to create a full-fledged AdbcConnection. This could be costly -- for example this may requiring re-running the auth workflow when an auth token was generated already or creating a session object instead of re-using the session already established.
An idea would be to have AdbcDatabase let the caller construct an AdbcPartitionedReader from a PartitionDescriptor instead of requiring the full connection process. The driver implementation can bake all the details on how to connect to and use the node holding the partition (including stateful information such as auth access tokens or session identifiers) so it can skip the potentially heavy-weight connection process.
Contributor guide
Research direction
Start by tracing the Java flow from AdbcStatement.executePartitioned() through PartitionedResult and PartitionDescriptor to AdbcConnection.readPartitioned(). Review how AdbcDatabase creates connections and how driver implementations retain connection state. Done means defining and implementing a way to construct a partition reader from a PartitionDescriptor without recreating the full connection workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100