apache / apache/arrow-adbc

feat: Add the ability to request a schema from a statement

Open
#1,514 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
627
Forks
217
Avg merge
22h 22m
Merged PRs (30d)
64

Description

There are some situations (e.g., #1513) where the mapping of a database type to an Arrow type is not canonical. SQLite is an example of an end-member where *all* mappings of a database result are approximate (and not necessarily stable between queries).

When I rewrote the typing part of the PostgreSQL driver, I intentionally separated the "guess Arrow type from Postgres type" and "convert Postgres data to Arrow data" components. Given an Arrow type, it's reasonably straightforward to write the conversion from a Postgres type. The hard (and imprecise) part is the guessing.

Instead of providing a possibly ever-accumulating pile of options along the lines of `"adbc.postgresql.statement.numeric_as_double" = "true"`, I wonder if we could add `AdbcStatementRequestSchema(struct AdbcStatement*, struct ArrowSchema*)`. Often the query author knows this information (or is using a SQL generation tool that already knows what column types to expect). In more dynamic wrappers, one could inspect `AdbcStatementExecuteSchema()` and look for specific types. This model fits nicely with how the Python `__arrow_c_stream__(requested_schema=xxxx)` protocol is parameterized as well.

I'm not sure whether the request should be best-effort or error-if-cannot-be-satisfied (or whether the caller should be able to choose). But without the ability to pass an `ArrowSchema*`, it's very difficult to work around this: you could provide an IPC-serialized schema to `AdbcStatementSetOptionBytes()`.

Contributor guide

Open the contributing guide

Research direction

Begin with the existing AdbcStatementExecuteSchema and AdbcStatementSetOptionBytes entry points, and compare the proposed AdbcStatementRequestSchema with Python's __arrow_c_stream__(requested_schema=xxxx) protocol. The work is done when the request API, best-effort or error semantics, caller choice, and corresponding coverage are agreed and implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql, python, sqlite
Domain
api, backend-api-design, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.