apache / apache/arrow-adbc

Postgresql driver does not support multidimensional arrays

Open
#3,621 4 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
C#
Stars
627
Forks
217
Avg merge
22h 22m
Merged PRs (30d)
64

Description

### What happened?

The driver does not seem to support multidimensional arrays:

- Reading multi dimensional array literal flattens them:

```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("SELECT '{{a,b},{b,c}}'::text[][] as x")
table = cur.fetch_arrow_table()
print(table)
assert table.to_pydict() == {'x': [[['a', 'b'], ['b', 'c']]]}
```

- Same happens when reading from a table:

```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("""CREATE TABLE text_table (text_data text[], text_data2 text[][])""")
cur.execute("INSERT INTO text_table VALUES ('{x,y}', '{{a,b},{c,d}}')")
cur.execute("SELECT text_data, text_data2 FROM text_table")
res = cur.fetchall()
print(res)
assert res == [(["x", "y"], [["a", "b"], ["c", "d"]])], res
```

- insert data into multidimensional array does not work

```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("""CREATE TABLE int_table (int_data int[], int_data2 int[][])""")
cur.execute("INSERT INTO int_table VALUES ($1, $2)", ([1,2 ], [[1,2], [2,3]]))
```

Similar erros happen with text types. Passing arrow.scalars does not help either, regardless of splicit typs. Same error with either of these lines:
- `cur.execute("INSERT INTO int_table VALUES ($1, $2)", (pa.scalar([1,2 ]), pa.scalar([[1,2], [2,3]])))`
- `cur.execute("INSERT INTO int_table VALUES ($1, $2)", (pa.scalar([1,2 ], pa.list_(pa.int32())), pa.scalar([[1,2], [2,3]], pa.list_(pa.list_(pa.int32())))))`

have not tried updates, but I guess a similar error would happen.

For my use care the most problematic issue is the fact that dimensionality is lost while reading multidimensional data without any error being reported

### Stack Trace

```
Traceback (most recent call last):
assert table.to_pydict() == {'x': [[['a', 'b'], ['b', 'c']]]}, table.to_pydict()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: {'x': [['a', 'b', 'b', 'c']]}
```

```
Traceback (most recent call last):
assert res == [(["x", "y"], [["a", "b"], ["c", "d"]])], res
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: [(['x', 'y'], ['a', 'b', 'c', 'd'])]
```

```
Traceback (most recent call last):
File "", line 279, in
cur.execute("INSERT INTO int_table VALUES ($1, $2)", ([1,2 ], [[1,2], [2,3]]))
File "C:\Users\csl\miniconda3\envs\mamba\Lib\site-packages\adbc_driver_manager\dbapi.py", line 751, in execute
handle, self._rowcount = _blocking_call(
^^^^^^^^^^^^^^^
File "adbc_driver_manager/_lib.pyx", line 1770, in adbc_driver_manager._lib._blocking_call
return func(*args, **kwargs)
File "adbc_driver_manager/_lib.pyx", line 1364, in adbc_driver_manager._lib.AdbcStatement.execute_query
check_error(status, &c_error)
File "adbc_driver_manager/_lib.pyx", line 261, in adbc_driver_manager._lib.check_error
raise convert_error(status, error)
adbc_driver_manager.InternalError: INTERNAL: nanoarrow call failed: PostgresType::FromSchema(type_resolver, bind_schema->children[i], &type, &na_error) = (22) Invalid argument. Postgres array type with child oid 1016 not found
```

trying with text raises a similar error:
> adbc_driver_manager.InternalError: INTERNAL: nanoarrow call failed: PostgresType::FromSchema(type_resolver, bind_schema->children[i], &type, &na_error) = (22) Invalid argument. Postgres array type with child oid 1009 not found

### How can we reproduce the bug?

See examples above.

### Environment/Setup

adbc_driver_manager = 1.8.0
adbc_driver_postgresql = 1.8.0

Contributor guide

Open the contributing guide

Research direction

Start with the PostgreSQL DB-API paths exercised by connect, cursor.execute, fetch_arrow_table, and fetchall, then inspect the PostgresType::FromSchema error for multidimensional bindings. Done means reads preserve nested array dimensions and inserts accept multidimensional integer and text arrays without the reported errors; reproduce the examples in the issue to verify both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.