apache / apache/arrow-adbc

Upgrade `adbc_ingest` parameter `temporary` to `prefixes`

Open
#2,343 14 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
C#
Stars
627
Forks
217
Avg merge
17h
Merged PRs (30d)
57

Description

### What feature or improvement would you like to see?

### Describe the use case

Currently `adbc_ingest` supports a parameter `temporary` to execute `CREATE TEMPORARY TABLE` statements. `sqlalchemy` supports this in a more abstract way, by accepting a list of table [`prefixes`](https://docs.sqlalchemy.org/en/20/core/metadata.html#sqlalchemy.schema.Table.params.prefixes). I suggest to upgrade the API by implementing a `prefixes` parameter like `sqlalchemy.Table`.

### Databases / Backends / Drivers targeted

Some DBMS support the use of multiple prefixes. E.g.
`ibm db2`: [`CREATE GLOBAL TEMPORARY TABLE`](https://www.ibm.com/docs/en/db2/11.5?topic=statements-create-global-temporary-table)
`postgres`: [`CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE`](https://www.postgresql.org/docs/current/sql-createtable.html#SQL-CREATETABLE-TEMPORARY)
`snowflake`: [`CREATE [ { [ { LOCAL | GLOBAL } ] TEMP | TEMPORARY | VOLATILE | TRANSIENT } ]
TABLE`](https://docs.snowflake.com/en/sql-reference/sql/create-table#syntax)

### Example Use

Create a global temporary table from some data.

```python
with con.cursor() as cur:
total_inserted = cur.adbc_ingest(
table_name="TEST",
data=some_data,
mode="create",
db_schema_name="SESSION",
temporary=["GLOBAL", "TEMPORARY"],
)

con.commit()
```
User can then use this "empty shell" to insert and further process data within their respective session.

### Additional context

I'm currently working on adding a `prefixes` parameter to `pandas to_sql()` method, see https://github.com/pandas-dev/pandas/pull/60409 for reference. Since `to_sql()` supports both `sqlalchmy` and `adbc` it would be beneficial for the ecosystem to have feature parity. The current state of progress:

- I fully implemented `prefixes` for `sqlalchemy` drivers
- I implemented a way to only use the `TEMPORARY` keyword from `prefixes` for `adbc` drivers

Contributor guide

Open the contributing guide

Research direction

Start at the adbc_ingest API's existing temporary handling and compare it with SQLAlchemy Table prefixes and the referenced pandas to_sql work. Check how the targeted database drivers construct CREATE TABLE statements. Done means prefixes accepts multiple values and provides the requested temporary-table behavior across the relevant ADBC drivers.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, sqlalchemy
Domain
api, backend, database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.